Ryan Harrison My blog, portfolio and technology related ramblings

Git - Remove Passwords or Sensitive Data

If you have accidentally committed a password or other sensitive data to your Git repository, make sure to remove it before pushing it to GitHub or making it open source. When working with databases or other external systems, it’s incredibly easy to put a password into a configuration file, commit it and forget about it - only for it to be subsequently completely visible to everyone on the internet thereafter.

Thankfully, there are a few ways to remove private data from your Git repos. The most bare-metal approach would be to use the git-filter-branch command which is able to rewrite your Git history whilst applying filters. It’s a very powerful command and therefore has many configuration options and is quite difficult to use directly.

A much easier solution is to use the BFG Repo Cleaner. This is a separate tool which offers a much faster and simpler method of cleaning your repo of private data or very large files.

To make use of this tool, first make sure you have Java installed and available on the PATH - follow the steps in this post to do so. Then download the .jar file from their site.

Find sensitive data within your repository

If you have a password or other credential that you think you may have accidentally committed into your repo, run the following command to find out. This uses the ‘pickaxe’ option of git log to find revisions in which the number of occurences of a search time has changed.

$ git log -S<password>

From the docs:

  • -S Look for differences that change the number of occurrences of the specified string (i.e. addition/deletion) in a file.
  • -G: looks for differences whose added or removed line matches the given regexp, as opposed to -S, which “looks for differences that introduce or remove an instance of string”.
  • –all: searches over all branches and tags

Remove passwords and private data using BFG

  1. Create a file called passwords.txt, in which place all your passwords or sensitive data.
  2. Run the command $ java -jar bfg.jar --replace-text passwords.txt <your-repo>

More documentation and examples, along with how to remove large files from your repo can found on the BFG project page.

Note: This will completely rewrite your Git history, the hashes or all commits will be re-generated.

Read More

Visual Studio Code - Markdown Preview

Visual Studio Code has now replaced Sublime Text as my editor of choice due to it’s speed and first class support for a number of features that were previously only available through extensions (of varying quality and often outdated). One such example is a live preview of a markdown page - something essential when writing blog posts or pretty much any documentation these days.

Thankfully, Visual Studio Code not only provides native support for Markdown, but also a great live preview of the results as you type - all without any extensions. To enable the live preview within a new editor tab, use the keyboard shortcut Ctrl+Shift+V when any Markdown (.md) file is open. This works, but it’s much better to have the live preview of the HTML in a split editor so you can see it as you type. VS Code allows you to do this through the shortcut Ctrl+K V. Alternatively, for either of these options you can go through the Command Palette Ctrl+Shift+P and select either of the ‘Markdown Preview’ options.

Jekyll WSL

As of version 1.9, the markdown preview and editor window both scroll in sync. The preview window now loads in local files and you can also now double click on any element within the preview to jump directly to the corresponding line in the editor tab.

Spelling and grammar checking isn’t currently available within Visual Studio Code out of the box (hopefully this will also be added in a future update). There is however a great extension available on the marketplace which does a great job (when the API it relies on is up).

Read More

Installing Jekyll using the WSL

Note: As of writing file watching is currently not implemented in the WSL so you won’t be able to use jekyll serve --watch. If you are on an insider preview #14942 you will have access to this feature (request page)

Thanks to the awesomeness that is the WSL Windows Subsystem for Linux, you can now install and use a load of tools on Windows just as you would do under Linux. Previously I used to run Jekyll (running under Ruby) solely through a Unix based environment - creating new blog posts through a system of SSH and SFTP. Ruby development on Windows has never really been that great, but now the whole process has got much easier - run everything locally through the WSL layer.

The first step is to install Ruby. Following the instructions from their site:

1. Install dependencies:

sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev nodejs

2. Install Ruby via rvm:

I had troubles installing Ruby via rbenv, but rvm seems to work just fine.

curl -sSL https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
rvm install 2.4.0
rvm use 2.4.0 --default
ruby -v

(you will be prompted to enter your password a couple times for apt-get installs of requirements)

3. Install Bundler:

gem install bundler

4. Install the Jekyll Gems:

gem install jekyll
gem install jekyll-sitemap
gem install jekyll-paginate

5. Install the Pygments highlighter

(not needed if you are using the Rouge highlighter) Make sure you have Python and Pip installed first then:

sudo pip install pygments
gem install pygments.rb

6. Confirm the installation

jekyll -v

You should see output like

jekyll 3.3.1

7. Build and serve your site

You can’t use the watch command via file watchers yet (see note at the top), but you can use polling to get near the same effect:

jekyll serve --force_polling --incremental

Jekyll WSL

Read More

WSL - Creating a shortcut to Windows Documents

Due to the fact that you should never change Linux files using Windows apps and tools when running the WSL, I find myself during the vast majority of work in my Windows documents area and then accessing them directly via Bash.

Your Windows drives are accessible via /mnt using the same drive letters as found in Windows explorer - e.g to access your Windows Documents area via Bash you can cd to:


This works just fine, but it’s a bit long winded when most of your frequently accessed files are located there (plus it looks ugly having such a long path in your prompt all the time). Thankfully, the process gets much simpler through the use of symlinks. Using the following command you can create a symlink pointing directly to your Windows Documents folder:

ln -s /mnt/c/Users/<username>/Documents ~/docs

You can then just do a quick cd ~/docs and end up right in your Windows home directory. The great part is that your prompt will also just show ~/docs instead of the full path via /mnt. I have a number of these symlinks set up, pointing to various places in my Windows drive, and they make the whole experience when using Bash much more fluid.

Read More

Ubuntu - Clear Disk Cache

The Linux kernel often uses available memory for disk caching unless its otherwise needed by a program running on the system. This feature can speed up the system, but the RAM will be marked as used in the top command. It might seem that you are out of memory and need to do something to resolve the problem, when in fact everything is normal and the kernel will simply give programs resources from the disk cache when required. It never takes resources away from programs, just makes use of idle memory sitting in your system. This is such a common question that it even has its own website - http://www.linuxatemyram.com/.

If you really want to clear the disk cache and free the memory, run the following command:

sudo sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches

Note: there isn’t much point though, as the kernel will just start using the memory again for disk caching straight after you free it.

Read More