Ryan Harrison

Mocking Time in Java

Wed, 02 Sep 2026 00:00:00 +0000

If your code calls LocalDateTime.now() or Instant.now() directly, you’re stuck with whatever the system clock says. This makes it nearly impossible to write deterministic tests for time-sensitive logic like expiration checks, scheduling, or time-based calculations.

The solution is to inject time as a dependency rather than calling static now() methods directly. Java’s Clock abstraction provides a clean way to do this, and there are several approaches depending on your needs.

The Problem with Static now() Calls

Consider a service that checks if a subscription has expired:

public class SubscriptionService {
    public boolean isExpired(Subscription subscription) {
        return subscription.getExpiryDate().isBefore(LocalDateTime.now());
    }
}

This code is difficult to test because LocalDateTime.now() always returns the actual current time. You can’t easily test the boundary conditions around expiration without changing your system clock or waiting for time to pass.

Using Clock for Dependency Injection

The java.time.Clock class was designed to solve this problem. Instead of calling static now() methods, you pass a Clock instance to the time API:

LocalDateTime.now(clock);
Instant.now(clock);
ZonedDateTime.now(clock);

Refactor your code to accept a Clock parameter:

public class SubscriptionService {
    private final Clock clock;

    public SubscriptionService(Clock clock) {
        this.clock = clock;
    }

    public boolean isExpired(Subscription subscription) {
        return subscription.getExpiryDate().isBefore(LocalDateTime.now(clock));
    }
}

In production, inject Clock.systemDefaultZone() or Clock.systemUTC(). In tests, use a fixed clock.

Built-in Clock Implementations

Java provides several Clock implementations out of the box:

Clock.fixed()

Creates a clock that always returns the same instant. Good for most unit tests:

@Test
void testExpiredSubscription() {
    Clock clock = Clock.fixed(
            Instant.parse("2026-01-20T10:00:00Z"),
            ZoneId.of("UTC")
    );

    SubscriptionService service = new SubscriptionService(clock);

    Subscription subscription = new Subscription(
            LocalDateTime.parse("2026-01-20T09:00:00")
    );

    assertTrue(service.isExpired(subscription));
}

Clock.offset()

Creates a clock that’s offset from another clock by a duration. Useful when you need to test time progression:

@Test
void testRenewalWindow() {
    Clock baseClock = Clock.fixed(
            Instant.parse("2026-01-20T10:00:00Z"),
            ZoneId.of("UTC")
    );

    // Create a clock 30 days in the future
    Clock futureClock = Clock.offset(baseClock, Duration.ofDays(30));

    RenewalService service = new RenewalService(futureClock);
    assertTrue(service.isInRenewalWindow(subscription));
}

Clock.system()

The production clocks that return the actual system time:

// Uses system default timezone
Clock.systemDefaultZone();

// Uses UTC timezone
Clock.systemUTC();

// Uses specific timezone
Clock.system(ZoneId.of("America/New_York"));

MutableClock from ThreeTen-Extra

For tests that need to simulate time progression, the MutableClock from the threeten-extra library is useful. It allows you to advance time during a test.

Add the dependency:

<dependency>
    <groupId>org.threeten</groupId>
    <artifactId>threeten-extra</artifactId>
    <version>1.8.0</version>
    <scope>test</scope>
</dependency>

Use it to control time in your tests:

@Test
void testTokenExpiration() {
    MutableClock clock = MutableClock.of(
            Instant.parse("2026-01-20T10:00:00Z"),
            ZoneId.of("UTC")
    );

    TokenService service = new TokenService(clock);
    String token = service.createToken("user123");

    assertTrue(service.isValid(token));

    // Advance time by 2 hours
    clock.add(Duration.ofHours(2));

    // Token should now be expired
    assertFalse(service.isValid(token));
}

This is cleaner than creating multiple fixed clocks or using offset clocks for complex time-based scenarios.

Using Mockito MockedStatic (Antipattern)

Mockito 3.4+ supports mocking static methods with MockedStatic. You might be tempted to use this for mocking LocalDateTime.now():

@Test
void testWithMockedStatic() {
    try (MockedStatic<LocalDateTime> mock = mockStatic(LocalDateTime.class)) {
        LocalDateTime fixedTime = LocalDateTime.parse("2026-01-20T10:00:00");
        mock.when(LocalDateTime::now).thenReturn(fixedTime);

        SubscriptionService service = new SubscriptionService();
        // Test code here
    }
}

This is generally an antipattern and should be avoided:

Thread safety issues: MockedStatic affects all threads, which can cause flaky tests in parallel test suites
Design smell: If you need to mock static methods, it usually indicates your code isn’t designed for testability
Performance: Static mocking is slower than dependency injection
Scope: Easy to accidentally leak mocks between tests if not careful with cleanup

The Clock abstraction exists specifically to avoid needing static mocking. Use it instead.

Git - Disable autocrlf

Tue, 21 Apr 2026 00:00:00 +0000

If you’re on Windows, Git has almost certainly been silently messing with your line endings since the day you installed it.

TLDR: The fix is this one command:

git config --global core.autocrlf false

It’s on by default. It’s a terrible default. Mainly due to historical reasons that no longer exist.

What `core.autocrlf` Does

In short, Windows has historically used CRLF (\r\n) as its line ending character, while Linux and macOS use plain LF (\n). At some point, someone decided that Git on Windows should helpfully bridge this gap automatically. So by default, core.autocrlf is set to true on Windows, which means:

On checkout - Git converts LF line endings in the repository to CRLF in your working directory
On commit - Git converts CRLF back to LF before storing the file

In theory, this means your files always have Windows-style line endings locally, and the repository stays clean with Unix-style endings. Sounds reasonable enough. In practice though, it can be really quite annoying.

Why It’s a Bad Default

WSL Makes It Worse

If you use WSL (and you definitely should if you use Windows), this setting can cause issues. Your Linux environment expects LF line endings. Your Windows Git has helpfully checked out everything with CRLF. The result is that your scripts and config files have a stray \r at the end of every line, which causes failures. WSL and Git will also start warning you about line ending inconsistencies whenever you touch a repo:

warning: LF will be replaced by CRLF in some-file.sh.
The file will have its original line endings in your working directory

You’ve probably seen this warning before, or rather enough times that you have started ignoring it.

Modern Editors Handle LF Just Fine

The argument for CRLF on Windows was that Notepad couldn’t handle LF-only files - which was true in 2003. Notepad has supported LF line endings since Windows 10 (2018). Every other editor - VS Code, Sublime, Vim, IntelliJ, Notepad++ etc - has handled it just fine for even longer. There is no editor you are realistically using today that will break on LF line endings.

The Conversion Is Lossy in Practice

Even when it works as intended, the automatic conversion can cause issues. Binary files can get corrupted if Git misidentifies them as text. Files can show up as changed in git diff or git status purely because of line ending differences, with no actual content change.

If you work in a team where some people have autocrlf true and others have it false, you’ll end up with commits that are just line ending changes.

Disable it

Set it to false like it should be by default:

git config --global core.autocrlf false

From then on, Git leaves your line endings exactly as they are - no silent conversions on checkout, no conversions on commit. If you want to be explicit about line endings across a team, a .gitattributes file in the repository is a much better approach since it applies the same rules for everyone regardless of their local config.

Use LF everywhere, configure your editor to save with LF by default. Don’t think about it again (until your move machine).

Engineering Guidelines Site

Sun, 08 Mar 2026 00:00:00 +0000

I’ve put together a reference site for engineering best practices, available at guidelines.ryanharrison.co.uk. The goal is a single place to find guidelines and conventions covering the full stack - from core principles through to deployment.

Rather than being prescriptive about specific tools, most sections aim to explain the reasoning behind a recommendation so you can apply it to your own context.

What’s Covered

Principles and practices - Core engineering principles covers the fundamentals: SOLID, DRY, KISS, YAGNI, clean code, and when to apply them. There’s also a code review guide and sections on technical debt, pull requests, and git workflow.

Architecture - Patterns for microservices, event-driven systems, and multi-tenancy.

API design - REST fundamentals and patterns, GraphQL, and OpenAPI contract-first development.

Testing - The testing strategy page covers the overall approach. From there, individual pages go into unit, integration, contract, end-to-end, mutation, and chaos testing.

Security and observability - Security overview covering authentication, authorisation, input validation, and data protection. Observability covers structured logging, metrics, tracing, and alerting.

Languages and frameworks - Guidelines for Java, Kotlin, TypeScript, and Swift, with framework-specific sections for Spring Boot, React, Angular, React Native, Android, and iOS.

Infrastructure - Docker, Kubernetes, Terraform, and a fairly detailed AWS section covering compute, networking, storage, EKS, and more.

Building Jekyll Sites with Docker

Sun, 22 Feb 2026 00:00:00 +0000

This site is built with Jekyll. That unfortunately often means wrestling with Ruby versions, gem dependencies, and environment configuration. Different machines require different setups, and what works on your machine might not work in CI or on your VPS. Docker solves this for many other areas by providing a standalone and reproducible build environment, so why not here as well?

The challenge is that the official Jekyll Docker images are no longer being actively maintained. Thankfully, there’s a decent community alternative that handles modern Jekyll sites without the maintenance burden.

The bretfisher/jekyll-serve image is well-maintained (for now) and works as a drop-in replacement for the official Jekyll images. By default, it serves your site locally via jekyll serve, but you can override the command to run any Jekyll operation you need.

The image handles all the Ruby and gem setup for you, so you don’t need to worry about version conflicts or system dependencies.

Local Development with Docker Compose

For local development, the fastest approach is using Docker Compose. Create a compose.yml file in your Jekyll project root:

services:
  jekyll:
    image: bretfisher/jekyll-serve
    volumes:
      - .:/site
    ports:
      - "4000:4000"

Start your development server with:

docker compose up

Your site will be available at http://localhost:4000 with live reload enabled. The key benefit here is that Docker Compose reuses the same container across runs, which caches your gems. This means subsequent starts are very quick - typically just a few seconds once the gems are installed.

Without compose, you can achieve the same result with:

docker run -p 4000:4000 -v $(pwd):/site bretfisher/jekyll-serve

However, this creates a new container each time, which means reinstalling gems on every run (unless you mess around with more volume mounts, see below).

Building for CI/CD

For continuous integration or deployment builds, you want to build the static site without running the server. Override the default command to run the Jekyll build process:

docker run -v $(pwd):/site bretfisher/jekyll-serve bundle exec jekyll build

This generates your static site into the _site directory. Here’s an example GitHub Actions workflow that builds on every push (also the one used to build this site):

name: Build Jekyll Site
on:
  push:
    branches: [ master ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Build with Jekyll
        run: |
          docker run -v $:/site \
            bretfisher/jekyll-serve bundle exec jekyll build

The build is completely reproducible because the Docker image contains a known-good version of Ruby and all the necessary build tools.

Building on Low Memory Machines

If you’re like me and building Jekyll sites on machines with limited memory (1GB or less), you might run into issues. Gem installation often requires building native extensions, which can be memory-intensive. The bretfisher/jekyll-serve image runs bundle install --retry 5 --jobs 20 by default, which parallelizes gem installation, but uses more memory. This caused issues for me on resource-restricted boxes.

You can override the entrypoint to reduce the number of parallel jobs:

docker run -v $(pwd):/site \
  --entrypoint /bin/bash \
  bretfisher/jekyll-serve \
  -c "bundle install --jobs 2 && bundle exec jekyll build"

Reducing --jobs from 20 to 2 significantly reduces memory usage during the gem installation phase. The build will take a bit longer, but it won’t crash on memory-constrained systems.

Caching Gems for Faster Builds

If you’re building on a persistent machine like a VPS, you can use a volume mount to cache gems across builds:

docker run -v $(pwd):/site \
  -v jekyll-gems:/usr/local/bundle \
  bretfisher/jekyll-serve bundle exec jekyll build

The named volume jekyll-gems persists the installed gems. The first build installs all gems, but subsequent builds skip gem installation entirely and jump straight to building your site. This is much faster and significantly less resource-intensive.

You can also apply this technique in docker-compose:

services:
  jekyll:
    image: bretfisher/jekyll-serve
    volumes:
      - .:/site
      - jekyll-gems:/usr/local/bundle
    ports:
      - "4000:4000"

volumes:
  jekyll-gems:

That’s pretty much it. Easy enough Jekyll builds without messing around with Ruby/gem packages/versions.

Remote Debugging Java Apps with IntelliJ

Mon, 19 Jan 2026 00:00:00 +0000

A junior developer recently came to me for help with one of those classic issues which appears on a remote dev environment, but for whatever reason can’t (at least easily) be replicated locally. Their immediate thought was to add more log statements and redeploy the app to see what’s going on. Not unreasonable, but this is a dev environment, so I connected IntelliJ to one of the running containers and began stepping through the code and inspecting variables. They looked at me thinking I was performing some kind of black magic, so here’s an intro or a quick reminder of something which goes very underappreciated.

Why Remote Debug?

There are a few general scenarios when you might reach for remote debugging:

Environment-specific bugs - Issues that only appear in staging, testing, or production-like environments with specific configurations, data, or network conditions
Container debugging - When your application runs inside Docker containers or Kubernetes pods and you need to debug without rebuilding images
Shared development environments - Debugging applications running on shared development servers or VMs
Integration testing - Troubleshooting complex integration scenarios with external systems that can’t be replicated locally

Rather than relying solely on log statements or trying to recreate production conditions locally, remote debugging lets you step through the actual running code in the target environment - a lot better than adding log statements!

How It Works

Java remote debugging uses the Java Debug Wire Protocol (JDWP), which is a communication protocol between a debugger and a Java Virtual Machine. The JVM opens a socket that a debugger can connect to, allowing it to control execution, set breakpoints, inspect variables, and evaluate expressions.

The JVM can act as either a server (listening for debugger connections) or a client (connecting to a debugger). In most cases, you’ll configure the JVM as a server and then have your IDE connect to it.

Enabling Remote Debugging

To enable remote debugging, you need to pass specific JVM arguments when starting your Java application. The modern syntax (Java 9+) looks like this:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar myapp.jar

Here’s a high-level breakdown of what each parameter does:

-agentlib:jdwp - Loads the JDWP agent library for debugging
transport=dt_socket - Uses socket transport for the debug connection (the standard approach)
server=y - Configures the JVM to listen for debugger connections rather than connecting out to a debugger
suspend=n - Starts the application immediately without waiting for a debugger to attach. Use suspend=y if you need to debug startup code
address=*:5005 - Binds to all network interfaces on port 5005. You can specify a specific IP address or hostname instead of *

For older Java versions (Java 8 and earlier, so I hope you won’t see this), you might see the older syntax:

java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -jar myapp.jar

This accomplishes the same thing but uses deprecated flags. Note that in Java 8, the address parameter only accepts a port number, not the *:port format.

Common Config Scenarios

Docker Containers

When running Java applications in Docker, expose the debug port and configure the JVM arguments:

# Dockerfile
FROM eclipse-temurin:25
COPY target/myapp.jar /app/myapp.jar
EXPOSE 8080 5005
ENTRYPOINT ["java", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005", "-jar", "/app/myapp.jar"]

Then map the port when running:

docker run -p 8080:8080 -p 5005:5005 myapp

Spring Boot

For Spring Boot apps, you can pass the debug arguments through Maven or Gradle:

# Maven
mvn spring-boot:run -Dspring-boot.run.jvmArguments="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"

# Gradle
gradle bootRun --debug-jvm

The Gradle --debug-jvm flag automatically configures remote debugging on port 5005 with suspend=y, so the application will wait for a debugger before starting.

Environment Variables

Many application servers and deployment platforms let you set JVM options through environment variables:

# Generic Java applications
export JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"

# Spring Boot with environment variable
export JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"

The JAVA_TOOL_OPTIONS var is automatically picked up by the JVM without needing to explicitly pass it to the java command.

Connecting from IntelliJ IDEA

Once your Java application is running with remote debugging enabled, you want to actually step through and inspect things. This basically means you need to attach IntelliJ to it:

Create a Remote JVM Debug Configuration
- Go to Run → Edit Configurations
- Click the + button and select Remote JVM Debug
- Give it a descriptive name like “Remote Debug - Staging”
Configure the Connection
- Debugger mode: Select Attach to remote JVM (the default)
- Host: Enter the hostname or IP address of the machine running your Java application (use localhost if debugging a local Docker container)
- Port: Enter the debug port (typically 5005)
- Command line arguments: IntelliJ shows you the JVM arguments needed - useful for reference but not required here
Set Module Classpath
- Under “Use module classpath”, select the appropriate module
- This ensures IntelliJ can map the remote code to your local source files
Start Debugging
- Click Debug or press Shift + F9
- Set breakpoints in your code
- Trigger the code path in the remote application (make an HTTP request, send a message, etc.)
- IntelliJ will pause execution when breakpoints are hit

When the debugger attaches successfully, you’ll see a message in the Debug console: "Connected to the target VM, address: ' hostname:5005', transport: 'socket'".

Some Considerations

Source Code Synchronization

Remote debugging requires that your local source code matches the bytecode running remotely. If the versions don’t match, you’ll see incorrect line numbers, unexpected behavior, or the debugger won’t stop at breakpoints.

Performance Impact

Remote debugging does have a performance overhead, particularly when:

Stepping through code line by line
Evaluating complex expressions
Inspecting large objects or collections
Using conditional breakpoints

AKA don’t do it in production.

Security

Opening a debug port exposes your application to potential security risks. Anyone who can connect to the debug port has full control over the JVM, including the ability to execute arbitrary code and access sensitive data.

Best practices:

Never expose debug ports directly to the internet
Use address=localhost:5005 instead of address=*:5005 when possible, then use SSH tunneling to connect
Only enable remote debugging in non-production environments, or very temporarily in production with strict access controls
Use firewalls and security groups to restrict access to the debug port
Disable debugging after troubleshooting is complete

AKA don’t do it in production.

SSH Tunneling

For remote servers, SSH tunneling can provide a secure way to access the debug port:

ssh -L 5005:localhost:5005 user@remote-server

This forwards your local port 5005 to the remote server’s localhost:5005. Configure your IntelliJ debug connection to use localhost:5005, and the connection will be tunneled securely through SSH.

Oracle Cloud - Setting up a Server on the "Always Free" Tier

Sat, 28 Jan 2023 00:00:00 +0000

So it turns out that Oracle Cloud (OCI) offers an extremely generous free tier, and not many people seem to know about it. Beyond the 30 day free trial, which gives you $300 worth of free credits to use (not bad at all), they also offer a pretty substantial amount of infrastructure in their Always Free tier as well. In terms of server (VPS) infrastructure, this includes (at the time of writing at least):

2 AMD based Compute VMs with 1 vCPU (x86) and 1 GB memory each + 0.48Gbps max network bandwidth
4 Arm-based Ampere A1 cores and 24 GB of memory usable as up to 4 VMs + 4GBps max network bandwidth
2 Block Volumes Storage - 200 GB total
10 GB Object Storage – Standard + 10GB Object Storage Infrequent Acccess + 10GB Archive Storage
Outbound Data Transfer -10 TB per month

Now, whatever you may think of Oracle in general, you can’t deny that this is a good deal. You can in theory set up a max of 6 VPS instances, all for free, on a commerical cloud environment. Even if the setup process might be a bit awkward compared to other providers, you can’t really complain too much. You would be spending a fairly significant amount on the equivalent amount of infrastruture from elsewhere like AWS or DigitalOcean.

In the rest of this post I will quickly run through the steps to setup a small VPS server running Nginx on the free Oracle Cloud tier. This is a standard VPS, just like you would find anywhere else, running Ubuntu Server 22.04. A Terraform provider is also available if you wanted, but for simplicity I will go through the web console.

Create an “Always Free” Oracle Cloud Account

Go to https://www.oracle.com/uk/cloud/free/#always-free and create an account as usual. You will have to provide credit card information (I presume to prevent misuse), but won’t be charged. For the first 30 days you will be able to play with $300 of credit if you want to, but after that time is up your account will revert automatically to the Always Free tier.

Create an Instance

Once signed in, select the Instances option on the main dashboard (yes the console isn’t the best for navigation). You should be presented with the following screen which will show all active instances. You can see that I’ve created a couple already:

Click on the Create Instance button to create a new VM. This is the standard VPS configuration/settings page:

Placement - controls which AZ the server is deployed to. Can be kept at default to let OCI choose the best. It should be noted that I’m also creating instances directly in the UK (London) region which is great for latency
Security - keep as default
Image and Shape - select the Image and type of server (amount of resources) you wish to deploy:

For this example we will go with Ubuntu Server 22.04 minimal as our OS. It works great on the limited amounts of CPU and RAM on these nodes as it unbundles a lot of of the default packages that you probably don’t need anyway (you can get them back if you need).

For shape, you can choose between AMD (2.0 GHz AMD EPY 7551 x86) vs ARM based (Ampere) CPU’s. You are limited to 1GB RAM max on the AMD shapes (makes sense since they are more expensive), but up to 24GB on the ARM cores. You get a max of 4Gbps of bandwidth on those ARM boxes well, which is very impressive for a free offering (though I haven’t benchmarked what you actually get). Here we will go with anAmpere based VM with 2 vCores and 6GB of memory (did I mention already that all this is free?):

Networking - can keep these as default to use your default root Virtual cloud network and subnet. Also make sure the option is checked to assign a public IPv4 address to your instance
Boot Volume - by default you will get a 50GB volume. You can increase this if you want to, up the max of 200GB allowed in the free tier

Create an SSH Key

In the Add SSH Keys section you can choose to automatically generate a keypair, but I prefer to create my own. There are plenty of tools for this, for now I will use PuttyGen to create a new Ed25519 keypair. Save both the private and public key locally for use later as usual. Don’t worry I’m not using this key

Then paste the public key into the corresponding box in the console. Oracle Cloud (OCI) will inject this public key into the .ssh/authorized_keys file for the main ubuntu user on the new VPS instance, thus allowing you to login through SSH.

You can look through the other configuration options as needed, but that should be good enough for now. Press Create and wait a couple minutes for the instance to be created. Startup times seem pretty reasonable on Oracle Cloud.

Update the default Security Group

As you might expect, by default the security group (or security list in Oracle Cloud) will block all traffic coming into your server by default apart from port 22 for SSH. This is good, but as we want to setup an Nginx web server on ours, we need to add a couple new ingress rules.

In the Virtual Cloud Networks section, navigate into your default VCN which was created by default. On the left, select the Security Lists option. There should be a single default entry for the VCN. Here we will add two new Ingress rules for port 80 and port 443. It should look something like the following after the changes:

Now all that needs to be done is to login to the instance using your private key saved from earlier. The default user is called ubuntu so if using standard ssh commands then something like ssh -i /path/to/private/key ubuntu@ipaddresss should get you in.

The ubuntu user has sudo access by default, so you can now start installing packages and using the instance for whatever you need.

Follow my other guide posts on how to setup an Ubuntu Server instance from scratch. For now we can just install Nginx to see our server running: sudo apt install nginx and sudo systemctl status nginx to check that it’s running.

Configuring iptables

One thing that might cause issues is the fact that the Oracle Ubuntu image sets up an iptables rule to block all traffic by default (ufw is not installed). That means if you had an Nginx server running, you won’t be able to ping it using the public IPV4 address unless you open up access on the ports. This seems a strange choice considering this is also controlled by the security group, but extra layers can’t hurt I guess.

To allow access on ports 80 and 443 for a standard web server with HTTPS enabled, run the following commands:

sudo iptables -I INPUT 6 -m state --state NEW -p tcp --dport 80 -j ACCEPT
sudo netfilter-persistent save
sudo iptables -I INPUT 6 -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo netfilter-persistent save

If you now navigate to the public IPv4 address in a browser, you should see the standard Nginx welcome page:

That about wraps up the setup process for this post. As I said, the Oracle Cloud free tier is extremely generous in terms of the sheer amount of infastructure you can provision. Plus, it operates just like the others big players as a proper commerical cloud offering, so uptime, network performance and integration with the general infra tooling should work out of the box.

Kafka vs MQ

Sat, 08 Oct 2022 00:00:00 +0000

Some quick high levels notes on Kafka vs MQ. This is a question that often gets asked by folks already who are familiar with traditional queues (IBM MQ/RabbitMQ etc) when they are introduced to the world of Kafka:

Kafka High Level Uses

Event input buffer for data analytics/ML
Event driven microservices
Bridge to cloud-native apps

ActiveMQ

Consumer gets pushed certain number of message by broker depending on prefetch
Consumer chunks through them, on each ack, broker deletes from data store
Produce pushes single message, consumer acks, deletes, gone 1to1
Conforms to standard JMS based messaging

Topics in MQ

Subscribers only receive messages published while it is connected
Or durable where client can disconnect and still receive messages after
In MQ can block brokers, fill data stores
Each consumer gets copy of the message unless composite destinations/message groups
Hard to create dynamic consumers or change the topology

Kafka

each group gets message, but in group only one consumer
consumers defines the interaction (pull)
- partitions assignment, offset resets, consumer group
consumer can apply backpressure or rebalance
Can’t go back through the log
Difficult to load balance effectively
Completing consumers vs one partition still processing whilst other is blocked
Hard to change topology or increase number of queues
Hard to handle slow/failing consumers
not predefining functionality to behave like a queue or topic, defined by consumers
- introduce new consumer groups adhoc to change how destination functions
- single consumer group = queue
- multi consumer groups = topic
- what offset to start from
one consumer group can fail and replay whilst another succeeds
MQ always queue one out at a time - not depending on consumers, Kafka behaviour changes on number of partitions/consumers

Useful Video Links

Kafka Recommendations & Best Practices

Sat, 18 Jun 2022 00:00:00 +0000

Most Kafka design and configuration choices are use case dependent and come with trade-offs, so it’s hard to define any best usage rules. However, below are some points of interest and general recommendations based on previous experience that you may want to consider or look more into:

Topics

Ensure all topics have a consistent naming scheme. If using a shared cluster, prefix with the system name. Kafka can have issues when there are both underscores _ and periods . in the topic name, so choose one or the other as a common separator
Determine the number of partitions the new topic should have, taking into account:
- volume of data - more partitions increases the upper-bound on the number of concurrent consumers processing messages
- it is easier to add new partitions to an existing topic in the future rather than remove partitions if there are too many already
- during a rebalance, all consumers will stop and be reassigned new partitions. If there are too many partitions/consumers then this can be slower process to coordinate and redistribute and may cause additional stress on the cluster. Don’t set the partition count too high as there are side effects
- as a general rule of thumb, keep the number of partitions an easily and evenly divisible by your expected number of consumers e.g having 12 partitions will evenly spread load across 3 or 4 consumers subscribed to it and processing messages in parallel
How long data on the topics should be kept in the cluster (retention) - as small as possible. For shared clusters this will be a max of 7 days in order to support the cluster being completely rebuilt over the weekend. Retention can be set on a per topic basis as needed
The topic replication factor will be likely be determined by the existing cluster configuration to ensure message durability. Likly will be set to 3 or 4 to survive broker failure or complete data centre failover
Don’t set compression at the topic level, prefer at the producer level. Look into other topic level configuration properties such as compaction as needed depending on specific use case
What the key of the topic will be depending on partitioning and source data - remember all messages with the same key will be sent to the same partition
What the value POJO should be - see below for more details

Messages

When creating a key, take into account data skew on the broker and partitions - ideally there will be even distribution between them all. Otherwise you may see issues if one consumer is processing all the messages with the others sitting idle
- if not key is provided, the cluster will use a round-robin approach to distribute messages - this is better than having a bad custom key
- the key should contain data elements with high cardinality to increase the probability of good data distribution (based on hashes)
- Do not include variable data elements in keys (e.g version numbers). The overall id is a good key property for example as it means all versions of that event will be sent to the same partition. There is no ordering guarantee across partitions so if you need one message to be processed after another, they must have the same key
Use strongly typed keys and values on topic - don’t use Strings as its harder to manage deserialization and versioning
Consider using a message wrapper to add metadata fields to all messages (or use headers)
- adds a number of fields to better support tracking across services source instance, cause, origin, run/batch UUID and unique message id’s
If the topic is to consume very high volumes of data, then try to avoid unnecessary duplication or large objects which are not needed to improve throughput and network traffic
Consider changes to the format of the messages over time e.g adding or removing fields
- this has to be synced between producers/consumers to ensure the messages can still be deserialized properly
- it is also a reason to use short retention periods and consumers will not need to process very old messages
- consider looking into Avro and schema registry to better manage this aspect

Producers

If your project is Spring based, use the provided KafkaTemplate classs which wraps the Apache producer and makes config/sending messages much easier
Specify acks=all in the producer configuration to force the partition leader to replicate messages to a certain number of followers (depending on min-insync replias) before ack’ing the messages and received
- this will increase latency between sending and receiving ack, but otherwise you may lose messages if a broker goes down and the message has not been replicated yet
- KafkaTemplate operates in an sync manner by default, so call .get() on returned future to block until ack is complete
Enable the idempotence property on the producer to esnure that the same message doesn’t get sent twice under some error conditions (adds tracking ids to the message to prevent duplicates). Note that depending on use case and consumer behaviour this may not be needed. Be aware that this also overrides other producer configs such as retries (infinite) and max inflight requests
Performance at the producer level is mainly driven by efficient micro-batching of messages
- batch size and linger.ms can set to delay sending until the producer has more messages to include in the batch
- this is a balancing act as increasing these will add higher latency to message delivery, but give more overall throughput
Consider setting compression on the producer to improve throughout and reduce load on storage, but might not be suitable for very low latency apps as introduces a decompression cost on the consumer side. It is not recommended to add this to the topic level which will cause the cluster to perform compression on the broker instead of the producer

Consumers

If your project is Spring based, use the KafkaListener and KafkaListenerContainerFactory classes to greatly simplify the otherwise complicated setup of the core Apache consumer classes and adds inegrations with other aspects of Spring Boot
- makes setting deserializers, concurrency, error handling etc easier
Kafka only provides ordering guarantees for messages in the same partition, so if your consumer needs to see one message after another, they must have the same key
- even then rebalancing may cause disturbances in this behaviour
To scale consumption, place consumers into he same consumer group to evenly spread partitions between instances
If the same listener is subscribed to multiple topics, it is recommended to place them in different consumer groups, otherwise rebalancing will impact processing in all when not necessary
Improve throughput by configuring min bytes and max wait times for batching - again this is a balancing act of throughput vs latency
Ensure all consumers are idempotent (able to consume the same message multiple times without any negative impact)
- Kafka has a number of ways to try and combat duplication, but it is strongly recommended to place checks in all consumers as this is still never guaranteed from Kafka
- Sending offset acknowledgements are a balancing act between at-least-once and at-most-once processing - the former being preferable in most cases
Ensure that consumer lag (difference in rate of production vs consumption) is properly traced via monitoring tools to detect general issues or slow consumers
By default Kafka will auto-commit offsets as a specific interval, but this alone introduces risk of data loss (commits ofsset before fully processed vs processed before committing offset and then crashing)
- Is is strongly recommended to disable auto commit and manage offset commits at the application level
- You can process on message at a time and then sync commit the offset after processing is complete -safest approach but more network traffic as less offset batching
- Process the messages as a batch in your service code and then sync commit all the offsets at the end - more throughput but more chance of duplicated effort
- By making all consumers idempotent, these concerns are removed entirely as long as the offsets are committed after processing is completed

Error Handling

Ensure that all exception scenarios are thought about and handled (deserialization, delivery failure, listener failure, broker down, timeouts, transient issues, error topic publish etc) and alerts are raised when necessary (but ideally the system should be able to recover itself from many of these problems)
Avoid creating configuration/consumers/producers manually, use factory methods in a core lib which should come setup already for error handling in a a consistent manner

Kafka Spring Consumer

There are a number of techniques to handle errors when consuming topics depending on use case - pausing the consumer, raise alerts and continue, error topics etc
A common approach is for each topic to have associated error topics (suffix .error app exceptions and .deserr for deserialization exceptions). Messages are automatically forwarded to these error topics when corresponding exceptions are thrown
- Error topics always have just one topic to maintain message order over time
- If the deserializer fails to convert your JSON to the corresponding POJO, the message will be placed immediately on the .deserr error topic and an alert raised (no point in retries)
- Key and value will be String versions of the original record to cover format issues
- Headers of the message include specific exception details including the origin offset and partition and a stacktrace - very useful for debugging
- If error topic publication fails, the record is not acknowledged and the container paused + an alert is generated - signals the cluster is likely having issues
- Take into account the multiple consumer groups may be processing messages from the same topic - include the consumer group name in the error topic name
If message fails during processing within a listener (general app exception)
- Can retry processing multiple times to cover transient issues - be careful about duplicating work
- If an exception is still thrown when processing the same message (app level issue such as db connection) then the message(s) are placed on the .error topic and the offsets are committed to continue consuming the next messages
- Headers of the message include exception details and stacktrace just like deserialization errors
Ensure monitoring is in place to detach any depth on error topics
- Likely resolution will be to reflow the messages after the underlying issue resolved - making consumer idempotent is very important here
Ensure transactionality is setup to rollback other changes (db) if the consumer fails

Kafka Spring Producer

All producers require ack from all broker replicas before committing. By default this is set to ‘all’ meaning that before proceeding a number of brokers need to ack all the messages - as defined by the min-insync-replicas config property at the broker level
If using KafkaTemplate make sure to call .get() on the future to block until the ack is received (by default it will continue immediately and you may not know about the issue). With this behaviour an exception will be thrown as you would expect if the production fails
Ensure transactionality is setup to rollback changes if errors occur - pullback messages sent to other topics etc

Kafka Streams

Similarly to general consumer error handling, but as Kafka streams spolits work into a number of parallel stream tasks, is a lot harder to manage centrally
Specify a deserialization exception handler - similar to Spring consumers, to publish messages onto error topic automatically if a record cannot be deserialized. Stream will move onto the next record
- Create all Serde’s using builders to setup deserializers consistently and ensure handlers are included by default
All potential exceptions must be handled within each stream task (map/filter etc) - if an exception is not caught it will crash the streams thread and force a rebalancing
- default exception handlers will just log the error and continue
- could wrap all operations in try/catch and handle accordingly
- or could wrap the base KStream/KTable etc to give consistent error handling and avoid duplicated code - ‘protected’ versions
If streams thread does do for some reason, an uncaught exception handler should be provided to generate a critiical alert. If all the streams threads die (potentially by all trying to process the same message), nothing will be consumed at all and the app will be stalled
Ensure monitoring is in place to ensure that the streams app as a whole is live and healthy

Misc

Although during local development you could directly connect to the main dev cluster, it is recommended for devs to instead create and use their own local clusters
- this gives a lot more control during dev for debugging/modifying properties without impacting others
- any new topic can be created or invalid messages sent without also being seent/consumed by the main services and causing unnecessary exceptions and alerts
Maintain clear documentation behind the reasons behind each topic and the config behind them - volume expectations etc
Create a Kafka topology diagram showing clearly the path(s) taken by messages between components - this is especially important the more services and topics are created
Create documentation showing exactly what keys/values are put onto each topic - useful for debugging and new joiners
Ensure all Kafka configuration for new producers/consumers is done in a single central place (likely a shared lib) to avoid duplication and chance of issues if placed everywhere
- list of broker hosts per environment, properties for Kerberos and auth etc
- maintain consistency in configuration for all producers/consumers regardless of where they are used - serialization, error handling, transactions etc
Ensure you have the appropriate disaster recovery plans in place to either recover from the cluster being down or failover
- Kafka is not meant as a data store, although unlikely plan for all the data to be lost at any point
- Determine how the application should react if the cluster is not available - brokers being down or transient network issues

Kafka Command Cheat Sheet

Tue, 05 Apr 2022 00:00:00 +0000

Environment Variables

Set path to binaries

export JAVA_HOME=/path/to/jdk

export PATH=$PATH:/path/to/kafka/common/bin

Add Kerberos / SASL config options (if required)

export KAFKA_OPTS="-Djava.security.krb5.conf/etc/krb5.conf -Dzookeeper.sasl.client.username=myuser -Djava.security.auth.login.config=/var/tmp/jaas.conf"

export KCONFIG=/var/tmp/kafka-client.properties

Where properties contains:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=myuser

Set common broker lists

export KBROKER=host:9092

export ZBROKER=host:2181

Topics

Create a new topic

kafka-topics.sh --zookeeper $ZBROKER --create --replication-factor 3 --partitions 4 --topic topic1

Describe an existing topic

kafka-topics.sh --zookeeper $ZBROKER --describe --topic topic1

List all topics in the cluster

kafka-topics.sh --zookeeper $ZBROKER --list

Delete a topic

kafka-topics.sh --zookeeper $ZBROKER --delete --topic topic1.*

Alter topic configuration

kafka-configs.sh --zookeeper $ZBROKER --alter --entity-type topics --entity-name topic1 --add-config retention.ms=2592000000

Consumers

Run a console consumer starting from the beginning of a topic

kafka-console-consumer.sh --bootstrap-server $KBROKER --topic topic1 --from-beginning --consumer-config $KCONFIG

Consumer which prints key and value for each message

kafka-console-consumer.sh --bootstrap-server $KBROKER --topic topic1 --from-beginning --property print.key=true --property key.separator="|" --consumer-config $KCONFIG

Create a consumer inside a specific group

kafka-console-consumer.sh --bootstrap-server $KBROKER --topic topic1 --group group1 --consumer-config $KCONFIG

Producers

Run a console producer pushing to a specific topic

kafka-console-producer.sh --broker-list $KBROKER --topic topic1 --producer-config $KCONFIG

Include a specific key in each published message

kafka-console-producer.sh --broker-list $KBROKER --topic topic1 --property "parse.key=true" --property "key.separator=|" --producer-config $KCONFIG

Consumer Groups

List all consumer groups in the cluster

kafka-consumer-groups.sh --bootstrap-server $KBROKER --list --command-config $KCONFIG

Describe offsets and status of a consumer group

kafka-consumer-groups.sh --bootstrap-server $KBROKER --group group1 --describe --command-config $KCONFIG

Delete a consumer group

kafka-consumer-groups.sh --bootstrap-server $KBROKER --group group1 --delete --command-config $KCONFIG

Reset offsets to the beginning for all topics

kafka-consumer-groups.sh --bootstrap-server $KBROKER --group group1 --reset-offsets --to-earliest --all-topics --execute --command-config $KCONFIG

Reset offsets to beginning for a specific topic

kafka-consumer-groups.sh --bootstrap-server $KBROKER --group group1 --reset-offsets --to-earliest --topic topic1 --execute --command-config $KCONFIG

Reset offsets to latest

kafka-consumer-groups.sh --bootstrap-server $KBROKER --group group1 --reset-offsets --to-latest --all-topics --execute --command-config $KCONFIG

Reset offsets to specific value

kafka-consumer-groups.sh --bootstrap-server $KBROKER --group group1 --reset-offsets --to-offset 10 --topic topic1:10 --execute --command-config $KCONFIG

Reset offsets by a specific shift amount

kafka-consumer-groups.sh --bootstrap-server $KBROKER --group group1 --reset-offsets --shift-by -10 --topic topic1 --execute --command-config $KCONFIG

ACL’s

Grant superuser access to all topics and groups

kafka-acls.sh --authorizer-properties zookeeper.connect=$ZBROKER --add --allow-principal User:myuser --operation ALL --topic '*' --group '*' --cluster

Zookeeper

Connect to the Zookeeper cluster

zookeeper-shell.sh $ZHOST:$ZPORT

Set Zookeeper ACL for superuser

ls /
setAcl / sasl:myuser:cdrqa,world:anyone:r
getAcl /

https://medium.com/@TimvanBaarsen/apache-kafka-cli-commands-cheat-sheet-a6f06eac01b

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/kafka_admin_cli.html

Ubuntu Server Setup Part 10 - Install Docker and Docker Compose

Thu, 24 Feb 2022 00:00:00 +0000

It’s likely that you will want to run containers of some kind on your server, for that we’ll be installing and using Docker. This consists of a couple different parts - the Docker Engine itself which runs in the background as a daemon process, the docker CLI commands which allow you to interact with the Engine, and finally docker-compose which is another tool for easily managing multiple containers. On Ubuntu based machines the recommended way of installation is to use the official Docker repository.

Install Prerequisites and Set Up the Repository

Before we do anything more, update your local apt repositories and install a few prerequisite packages which are required in later steps:

$ sudo apt-get update

$ sudo apt-get install ca-certificates curl gnupg

Next, we need to add the official GPG key provided by Docker for their repository:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Finally, we can add the official Docker repository, specifying that the releases must be signed by the GPG key we downloaded in the previous step. The command below will point to the stable repository, but you can use the nightly or test channels if you prefer:

$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker

Now we have the Docker repository and keys setup, we can pull down the latest packages again and directly install Docker directly through apt:

$ sudo apt-get update

$ sudo apt-get install docker-ce docker-ce-cli containerd.io

If everything completed successfully you should now have Docker Engine running in the background and the ability to run the docker CLI commands to run and manage images/containers. To verify things are working correctly, start an instance of the hello-world image:

$ sudo docker run hello-world

More details at https://docs.docker.com/engine/install/ubuntu/

Configuration Tips

Run Docker CLI as a non-root user

You may have noticed in the verify step above that we had to invoke the docker run command as root in order to successfully run a new container. This is required since the Docker daemon always runs as the root user and it binds to a Unix socket which is also owned by root. This is quite painful when interacting with the CLI however, so there is a workaround using groups:

# add a new group called 'docker' (it might already exist)
$ sudo groupadd docker

# add the current user to the docker group
$ sudo usermod -aG docker $USER

# after logging out and back in again you should be able to run docker without sudo
$ docker run hello-world

Note that this is not the same as running the daemon itself as a non-root user, so all the same security implications still remain when running containers.

Run Docker on startup

By default the daemon processes required to interact with Docker are not configured to start when the system boots. To rectify this we can instruct systemd to automatically start them for us:

$ sudo systemctl enable docker.service
$ sudo systemctl enable containerd.service

More details at https://docs.docker.com/engine/install/linux-postinstall/

Install Docker Compose

For reasons I don’t fully understand docker-compose doesn’t ship with the core Docker packages and so requires an extra installation step. Fortunately, it’s just a single binary so we can just directly download it into a location within the current PATH and start using it:

# download the latest release binary (replace the version from https://github.com/docker/compose/releases)
$ sudo curl -L "https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# add execute permissions
$ sudo chmod +x /usr/local/bin/docker-compose

# verify everything is working
$ docker-compose --version
Docker Compose version v2.2.3

More details at https://github.com/docker/compose

Ryan Harrison

Mocking Time in Java

The Problem with Static now() Calls

Using Clock for Dependency Injection

Built-in Clock Implementations

Clock.fixed()

Clock.offset()

Clock.system()

MutableClock from ThreeTen-Extra

Using Mockito MockedStatic (Antipattern)

More Reading

Git - Disable autocrlf

What core.autocrlf Does

Why It’s a Bad Default

WSL Makes It Worse

Modern Editors Handle LF Just Fine

The Conversion Is Lossy in Practice

Disable it

Further Reading

Engineering Guidelines Site

What’s Covered

Building Jekyll Sites with Docker

Local Development with Docker Compose

Building for CI/CD

Building on Low Memory Machines

Caching Gems for Faster Builds

More Reading

Remote Debugging Java Apps with IntelliJ

Why Remote Debug?

How It Works

Enabling Remote Debugging

Common Config Scenarios

Docker Containers

Spring Boot

Environment Variables

Connecting from IntelliJ IDEA

Some Considerations

Source Code Synchronization

Performance Impact

Security

SSH Tunneling

More Reading

Oracle Cloud - Setting up a Server on the "Always Free" Tier

Create an “Always Free” Oracle Cloud Account

Create an Instance

Create an SSH Key

Update the default Security Group

Login to the Instance

Configuring iptables

Kafka vs MQ

Kafka High Level Uses

ActiveMQ

Topics in MQ

Kafka

Useful Video Links

Kafka Recommendations & Best Practices

Topics

Messages

Producers

Consumers

Error Handling

Kafka Spring Consumer

Kafka Spring Producer

Kafka Streams

Misc

Kafka Command Cheat Sheet

Environment Variables

Topics

Consumers

Producers

Consumer Groups

ACL’s

Zookeeper

Ubuntu Server Setup Part 10 - Install Docker and Docker Compose

Install Prerequisites and Set Up the Repository

Install Docker

Configuration Tips

Install Docker Compose

What `core.autocrlf` Does