Joe Nyland

Send mail from Postfix on a BT Home broadband connection

Wed, 14 Dec 2016 00:00:00 +0000

In order to send email out from a BT Home Broadband connection, you must use the smarthost that BT provide. Failure to use this usually means that mail is rejected on the receiving mail server(s) as BT’s dynamic IP address ranges are blacklisted.

Postfix is my MTA of choice and I recently needed to setup a new box to send email out from my broadband connection. Unfortunately, it took longer than necessary to piece together all the bits of info I needed to successfully send email out, so I thought I would document it here.

First, you need to ensure that you have setup a BT Email account. Click here and create yourself an email address, if you’ve not already got one in your BT account. You need to make sure that you record a copy of the email and password that you create as this will be used by Postfix to authenticate with the BT smarthost.

The following steps have been tested on CentOS 7 and Ubuntu 14.04 LTS so they should be pretty portable. I will assume that you have Postfix installed on your system. If you need to install Postfix, Google it :blush:

NOTE: Run all of the following commands as root (sudo -i)

Let’s export some variables that will be used in the config. Replace the values below with the email address and password you just created above:

export BT_EMAIL_ADDRESS=my_bt_email_username@btinternet.com
export BT_EMAIL_PASSWORD=my_bt_email_password

Next we configure Postfix to use the BT smarthost and how it should authenticate with it:

postconf -e relayhost="[mail.btinternet.com]:587"
postconf -e smtp_sasl_auth_enable=yes
postconf -e smtp_sasl_password_maps="hash:/etc/postfix/sasl_passwd"
postconf -e smtp_sasl_security_options=noanonymous

Next up, we create or append to a password file which we will contain the credentials needed to authenticate with the smart host:

cat >> /etc/postfix/sasl_passwd <<EOF
[mail.btinternet.com]:587 ${BT_EMAIL_ADDRESS}:${BT_EMAIL_PASSWORD}
EOF
chown root:root /etc/postfix/sasl_passwd  # For security, root should own this file
chmod 0600 /etc/postfix/sasl_passwd       # Only allow owner read+write
postmap /etc/postfix/sasl_passwd          # Generate the hash DB that Postfix expects

Optional: Running CentOS? You will need to install an additional package:

yum install cyrus-sasl-plain -y

Apply the changes to Postfix:

service postfix start; postfix reload

Test! (Replace with your email at the end)

echo "Hi! This is a test from Postfix via BT." | sendmail some_external_email@example.com

If you don’t receive the test email, check mailq and the mail log file:

tail -f /var/log/mail.log # Ubuntu/Debian
tail -f /var/log/maillog  # CentOS

Debug a PHP app in a Docker container with Xdebug

Sun, 16 Oct 2016 00:00:00 +0000

Recently, I decided that I was going to learn more about Docker. I’ve heard so many good things about Docker over the past year or so, I figured I best take a look and decide for myself whether it was something I should integrate into my workflow. In the past 6 months, Docker for Mac has been released too, so this really sparked my interest.

I was pleasantly surprised by how easy it was to get started with Docker. It felt very much like starting to learn Git. It was so easy in fact, I began containerising all of my local development environments in order to expose myself to Docker more, allowing me to learn more as I went along.

In next to no time, I’d containerised most of the apps that I work on and I’d even built several multi-container environments with Docker Compose, which is a really cool tool! Unfortunately, I faced a small but annoying complication with Docker when I wanted to containerise a PHP app.

I’m a big fan of the JetBrains IDEs, so my IDE of choice when working with PHP apps is PhpStorm. One of the features of PhpStorm is the integrated debugger, allowing you to debug your apps right from within the code base.

Usually, when setting a up a LAMP development environment, I’ll setup Xdebug too. Xdebug uses the DBGp protocol to provide interactive debug sessions. For a PHP web app, roughly speaking the process is this:

Browser submits a request to the server (sometimes including a special GET parameter in the query string to start the debugging session)
The server (running PHP with the Xdebug module loaded) attempts to connect to the host running the IDE (usually port 9000 TCP) where an Xdebug client is listening for connections.
The IDE responds with commands that instruct Xdebug to continue execution of the code and commands to interrupt the execution of code at predefined breakpoints which are easily set my the developer right inside the editor window.

The configuration required for Xdebug is something like this:

[xdebug]
xdebug.remote_enable=On
xdebug.remote_autostart=On

Usually when developing, the LAMP stack is running in a local environment on localhost. This means that when the request is received by the web server (we’ll assume Apache here), Xdebug can simply connect back to localhost to the Xdebug client in the IDE.

When the LAMP stack is running in a container though, Docker’s network stack sits between the running processes inside the container and the host machine that’s running Docker Machine. This means that when I run a web server in a container, then open that site in my browser, the web server inside the container will see that the request originated from a host with an IP like 172.17.0.1 which is the IP address of the host (Mac in my case) on the virtual network stack that Docker runs on the host.

If we look back up to how Xdebug works, the Xdebug module will need to connect back to the IDE on 172.17.0.1, using port 9000. Sadly, Docker for Mac doesn’t currently allow access to the host on this IP address from within the container. This appears to be a known issue, mentioned here.

For example:

Run a simple demo app in a container:

docker run -it --rm --name php-demo -p 8080:80 php-web-app-debug-demo

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
[Sun Oct 16 11:16:37.528569 2016] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.10 (Debian) PHP/5.6.26 configured -- resuming normal operations
[Sun Oct 16 11:16:37.528660 2016] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

Then try to access it:

curl localhost:8080

We see the request logged from 172.17.0.1:

172.17.0.1 - - [16/Oct/2016:11:26:25 +0000] "GET / HTTP/1.1" 200 321 "-" "curl/7.49.1"

If we try and connect back to the host running the IDE with Xdebug client from inside the container:

telnet 172.17.0.1 9000

Trying 172.17.0.1...
telnet: Unable to connect to remote host: Connection refused

We’re left in a sticky situation: we need to connect back to the IDE running on the Docker host from Xdebug running inside the container, but there doesn’t seem to be a way to access it. There are a couple of networking options that are available for running a container under, but none of them seem to alleviate this issue.

I was able to find that whilst Docker containers are unable to connect back to the host through the NATed interface (172.17.x.x), they are able to communicate with the host using the LAN IP of the host, which is usually a DHCP address issued by your home or office router/network infrastructure. For example, my Mac is running on my home network and my router has issued it the address 192.168.1.66. From within a container, I can connect back to PhpStorm running the Xdebug client on port 9000 just fine!

telnet 192.168.1.66 9000

Trying 192.168.1.66...
Connected to 192.168.1.66.
Escape character is '^]'.

However, this isn’t ideal. The address above is different for everyone and in order for Xdebug inside the container to work, it needs to be told to connect back to the same IP that the request originated from, or it needs to be explicitly told what address to connect to. This configuration will be baked into the image but if I enter my current IP address, next time I launch a container from this instance I might have a different IP or someone else might launch a container from the same image and Xdebug won’t work for them until they find out their IP, drop into a shell in the container and change the Xdebug configuration, etc. This all detracts from one of the biggest advantages to using Docker: portability.

So far, the best solution I have been able to find is to use an arbitrary hostname in the Xdebug configuration and provide the value to this hostname on launching the container. This can be achieved using the following Xdebug configuration:

[xdebug]
xdebug.remote_enable=On
xdebug.remote_autostart=On
xdebug.remote_connect_back=Off
xdebug.remote_host=docker_host

Rebuild the image with the above config included, then launch the container like so:

# Replace 192.168.1.66 with your actual LAN IP address
docker run -it --rm --name php-demo -p 8080:80 --add-host="docker_host:192.168.1.66" php-web-app-debug-demo

With that in place, you should be able to set some breakpoints in yur PHP code, tell PhpStorm to listen for debug connections, then request the site in your browser and be good to go!

I’ve thrown together a quick demo of this. The repo holding the source can be found here and you can pull a Docker image containing a working Xdebug setup from the Docker Hub like so:

docker pull joenyland/php-web-app-debug-demo

Update

Since writing this post, I’ve found that the issue around connecting back to the Docker host isn’t possible out of the box.

The issue with the above workaround is that it’s dependent on a potentially ever changing IP address. Another issue is that you may not have LAN IP address, for example you’re not connected to any networks and are working offline.

The Docker for Mac network notes mention the known issues with Docker for Mac and in there, there’s a suggestion to add an alias address to the lo0 interface on your Mac. lo0 is a loopback interface that’s always available on your Mac, regardless of whether or not you’re connected to a WiFi network. If you’re not connected to a network, you can use the above workaround I’ve talked about, but instead of using the LAN address, use a new loopback alias address, which is created like so:

sudo ifconfig lo0 alias 10.200.10.1/24

Then create a container, passing in the new IP address you just added to the loopback interface:

docker run -it --rm --name php-demo -p 8080:80 --add-host="docker_host:10.200.10.1" joenyland/php-web-app-debug-demo

It’s still annoying that we can’t just use the xdebug.remote_connect_back, but it seems to be a known issue to the Docker for Mac team, so hopefully there may be a better solution in the future when the Docker networking implementation on Macs improves.

How to install a Ruby Gem from a custom source via Puppet

Fri, 20 May 2016 00:00:00 +0000

Here’s the situation: You’ve written a Ruby gem that you want to install on a server, but you don’t really have any need to share the gem with the world via RubyGems.org. It’s a custom gem with bespoke code that no one would benefit from. You’ve pushed the gem to your Gem server and you use Puppet to manage your servers.

Here’s a simple, example Puppet manifest, showing how to install gem via Puppet from a custom source:

# manifest.pp
node default {
  package { 'cautious-potato':
    provider => 'gem',
    source => 'http://mygemserver.com',
    ensure => '0.1.0'
  }
}

Then apply that manifest:

sudo puppet apply --test manifest.pp

…and you should see version 0.1.0 of the gem installed from your own Gem server!

It took me a while to figure this out and I only worked out how to do it whilst reading through the source of the Gem Package provider in Puppet. I hope this saves others from wasting too much time on working out how to do this, like I did!

GitHub Pages may soon support SSL!

Wed, 18 May 2016 00:00:00 +0000

I was checking out some settings on my GitHub account today and I noticed in the Security section, several messages like this:

Hmm… repo.pages_https_redirect_disabled …I don’t remember changing that setting. I don’t even know where that setting is! I reached out to the awesome GitHub support team and they tell me that they’ve been working on an “experimental” feature: :grinning:

Sorry for the audit log noise! An experimental feature was accidentally enabled for your repository…

So, it looks like we may be seeing SSL on GitHub Pages sometime soon!

I'm doing the Manchester to Blackpool charity bike ride

Fri, 13 May 2016 00:00:00 +0000

Recently, I’ve been trying to get back into my mountain biking. It’s been tough, because I’ve been out of it for around 12 months now. I gave up cycling all together last year, following a rough patch with my depression. I completely gave it up: sold all by gear, all my bikes and vowed that I would never cycle again …and I didn’t. Fast forward 10 months and I was managing my depression and the hunger to ride was building once again!

A week or so ago, an email was sent around the office at work asking if anyone was interested in doing the Manchester to Blackpool charity bike ride. I’m not a massively confident person and I have terrible trouble with my head telling me a million reasons why I wouldn’t be able to overcome a challenge, but I’ve got really good at managing that thought now. 18-24 months ago I would never have said that I’d have the courage to ride from Manchester to Blackpool; the negative thoughts would creep in and talk me out of it well before I managed to sign up for a ride like that. Now, though, I’ve been able to battle through that and sign myself up to the do the ride!

The bike ride is in aid of The Christie Charity and is a 60 mile ride from the Imperial War Museum in Trafford, Manchester to the South Promenade, Blackpool. Those 60 miles will take us through Leigh, Wigan, Preston, Warton and into Blackpool. Admittedly, I’m apprehensive, but I’m actually quite looking forward to it. It will be a challenge, but I’m looking forward to overcoming it and raising some money for The Christie Charity along the way! I will be joining “The Greenhouse” team from MediaCityUK who will be raising money for the charity too!

Please sponsor me!

All donations will go straight to The Christie Charity.

Update 10/07/2016 :tada: :dancer: :confetti_ball:

I've done it!!! 💪🚴😅 pic.twitter.com/Al4tHR0Lu6
— Joe Nyland (@JoeNyland) July 10, 2016

I completed the ride today! I raised £175 for The Christie Charity, which is well above the £50 target I set myself when I signed up for the ride. Thank you to everyone who donated. The money you have donated will all go to a good cause.

Homesick Bash completion

Sat, 07 May 2016 00:00:00 +0000

Your dotfiles hold a lot of stuff in them: You’ve most probably spent a while accumulating handy Bash functions, config and variables and that all gets stored in lots of hidden files that live in your $HOME directory. If you change machines maybe between home and work or you like to keep everything version controlled for peace of mind, or you just :heart: Git (like me), I recommend you check out Homesick by Josh Nicols.

I use it a hell of a lot for all of my dotfiles, but one thing that I feel it’s missing out-of-the-box is Bash completion. I mash my tab button all day and all night and when a command doesn’t respond to my furious tabbing, it makes me feel sad that I’m going to have to use a whole lotta energy to think what parameters I want to give a command and the path to a file, etc. It really is awful :wink:

So, I thought: “I use Homebrew all the time and it’s never cost me a penny, so why not give something back and write my own Bash completion?” That’s just what I did!

You can install with:

brew tap homebrew/completions # Tap the homebrew-completions repo
brew install homesick-completion

This is my first Bash completion script that I’ve written. Previous to this, I had no idea how packages allowed you to tab complete stuff so this was a really interesting exercise to find out how it’s done, whilst doing something useful at the same time!

You can find the repo on GitHub here. If you’ve git any ideas for improvements, feel free to submit a pull request, or if you get any problems, open up an issue.

I hope this helps someone out!

GitHub Pages with Git LFS

Fri, 29 Apr 2016 00:00:00 +0000

Recently, I decided to take another look at Git LFS. I was blown away by the simplicity of the extension to the already awesome Git VCS but also the ease of use of GitHub’s implementation of the feature.

Git LFS allows you to store large like binary files, assets and other files that you don’t want to clog up a Git repo and store them on a storage network like the one provided by GitHub. You can get a bit more of an understanding on what Git LFS is here. It’s really easy to install and setup - just follow the instructions on the site.

Previously, I’ve blogged about my site telling you all about how wonderful Jekyll is on GitHub Pages. When I found out how great Git LFS is, I naturally wanted to know if I could use Git LFS to store assets for my sites, such as Photoshop assets, large images, PDFs, etc. Through trial and error, I found that GitHub Pages does not allow you to serve assets from Git LFS in your GitHub Pages site, which is a little disappointing. :disappointed:

I turned to Uncle Google to make sure that there wasn’t a workaround for this that I hadn’t thought of, but all I found was this. So it seems (from a GitHub employee) that Git LFS is not supported by GitHub Pages, outright. Sad, because it would be great!

GitHub are a great company and in the past when I’ve had questions or queries, they’ve been most helpful. I decided to ask them why they don’t support this feature, merely to satisfy my own curiosity. Jess Hosman at GitHub kindly replied to my email (really quickly! :grinning:):

Hi Joe,

Thanks for getting in touch! LFS with GitHub Pages is something we’d love to support in the future, but still have some work to do to get us there. I’ll definitely pass this along to the team, and let them know you’d find that feature useful.

Thanks so much for the feedback, and let us know if you notice anything else that could be improved.

Cheers,

Jess

So, at least there’s hope that this feature will be supported by GitHub in the future - It wasn’t just a point blank “No!” that we come to expect from large companies like Apple and Google nowadays.

This post is a little different from my recent posts as it’s less of a tutorial/how-to format and instead a more “FYI” style post. Hopefully Google will index it and help others get to the conclusion I did, without having to waste too much time. Furthermore, hopefully GitHub will implement this feature sooner rather than later!

Running instances in a private subnet on Amazon EC2

Fri, 22 Apr 2016 00:00:00 +0000

For a while now, I’m been meaning to look into how best to setup EC2 instances on Amazon’s cloud computing platform AWS in a private subnet. In this post, I aim to explain how best to build a setup where your “frontend” web server(s) or load balancer(s) are situated in a publicly accessible subnet and your application nodes are in a private subnet that is not internet accessible.

Overview

This setup is just one example of a security best practice and there are many, many other steps that you should take to ensure the security of your stack. This particular practice ensures that the only part of your setup that is public facing is a thin, dumb layer. This layer is usually in the form of a load balancer but can just be a simple reverse proxying web server passing traffic between the untrusted public domain to the trusted domain of the private subnet where your application servers are situated.

The idea being: you must have an opening on to the internet, so let attackers do what they want with your public facing load balancer. If they want to kill it, DDOS it, or whatever - it’s a throw away instance that does nothing more than passing data between two interfaces and therefore can be terminated and a newly built (and security patched) instance can be put in it’s place without taking the application servers down. What’s more: an attacker does not have direct access to your app servers from the internet - they only get direct access to your load balancer or reverse proxy which is much more secure. Again, this doesn’t garauntee an attacker won’t hop from the load balancer onto the app server, but this can be mitigated with different users, SSH keys, etc. between the public subnet and the private subnet. This is a little out of the scope of this post, though!

Now, let’s get back on topic!

Amazon VPC

Amazon VPC is where we will carry out most of the work. It’s where you configure and manage you virtual network on AWS’ infrastructure and it’s the network where your EC2 instances will run.

VPCs

Login to AWS and load up the VPC Dashboard. Although it would be best to create a new VPC for the purposes of this post, we will be building our setup in the default VPC of your AWS account. If I’m not mistaken, AWS accounts automatically get a VPC setup for them out-of-the-box. Click “VPCs” in the VPC dashboard and make sure that you’ve got at least one VPC setup and ready to go!

If you’ve not got a VPC there, consult the AWS documentation on how to set one up.

Whilst I’m writing this post, I’m using a test VPC called test and I’ve configured to cover the IP range 10.0.0.0/16.

Internet gateway

Next, we need to create an internet gateway. Think of this as your internet connection at home plugging into the back of your home router.

In the left of the VPC interface, click “Internet Gateways”, then click “Create Internet Gateway”. We’ll call this test-gateway, because it’s well, a test! When that’s been created, it should show up with a state of detached. Select it from the list then click the “Attach to VPC” button above. Select your VPC from the dropdown and click “Yes, Attach”. In a few moments, your internet gateway should show as attached!

Subnets

Next up: Subnets! In the left of the AWS interface, under “Virtual Private Cloud”, hit “Subnets”. When the page loads, you should see all the subnets in all of your VPCs.

First, we’re going to create our public subnet. This subnet is where we will place our load balancer and will therefore be publically accessible (firewalled, of course). Hit “Create Subnet” and name this new subnet “public”. If you created a new VPC for testing, select it in the VPC field below. For the CIDR block, select a subnet in the range that your VPC covers. For example, I created a VPC covering 10.0.0.0/16. That means I’m free to use anything between 10.0.0.0 and 10.0.254.0 for my subnets. In this example, I will suggest that you use a CIDR block for this subnet of 10.0.0.0/24. You can leave the “Availability Zone” option on the default of “No Preference”.

When your public subnet has been created, hit “Create Subnet” again and we will create our private subnet where our application servers will run. Enter the name “private”, select your VPC and enter a CIDR block that does not overlap with any of your other subnets. In the example I gave above, we could use 10.0.1.0/24 for this subnet.

Route Tables

You should now have (at least) two subnets in your VPC: one “public” and one “private”. We now need to give the “public” subnet internet connectivity. Select the “public” subnet and click the “Route Table” tab. Below, the route table’s ID should be displayed and this should link you to the “Route Tables” configuration screen for this table in the VPC interface. Click that link to be taken there. Select the route table in the table of results, then in the view below, click the “Edit” button which should allow you to edit the route table.

Click the “Add another route” button and a new row in the table should show up. Fill in the details as follows:

Destination: 0.0.0.0/0
Target: test-gateway (As you enter this, you should be able to select the igw-a1b2c3 style ID of the gateway)

Click the “Save” button.

We now want to create a new route table for our private subnet. Click “Create Route Table” and call it “private”, making sure that you’ve selected the correct VPC for your account. In the “Route Tables” screen, you should now have at least two route tables and one of them should be called “private”. Select this route table from the list and then click “Subnet Associations” in the view below. Click “Edit” and then tick the box under “Associate” alongside the “private” subnet, which we created earlier. Hit the “Save” button to apply our changes.

NAT Gateway

In it’s current state, if we were to launch an EC2 instance in the “private” subnet, the instance would not be able to connect to the internet. This is great from a security perspective and you may want to be super secure, but it makes things hard when you want to deploy to the instance and part of that deploy pulls down code from a GitHub repository, or you want to install the latest OS updates and security patches.

To fix this, we need to setup a NAT Gateway to provide secure access to the outside world. In fact, this is just what your home router does. You may wish to read more on NAT and Amazon’s NAT Gateways if you want to understand what they do and how they work. Note that this is not the same as the Internet Gateway that we setup for our “public” subnet earlier! Whilst they provide internet connectivity, they do it in different ways.

With that explained, let’s proceed with the tutorial and click on “NAT Gateways” in the left hand side of the VPC interface. Click “Create NAT Gateway”. In the modal that pops up, enter your “public” subnet into the “Subnet” field and then click “Create NEW EIP” then “Create NAT Gateway”. You should get the following note displayed:

Note: In order to use your NAT gateway, ensure that you edit your route tables to include a route with a target of ‘nat-q1w2e3r4t5y6’.

Hit the “Edit Route Tables” button to be taken to your route tables screen. Select the route table we created earlier and then click the “Routes” tab below. Hit the “Edit” button, then click the “Add another route” button and a new row in the table should show up. Fill in the details as follows:

Destination: 0.0.0.0/0
Target: nat-q1w2e3r4t5y6 (As you begin to enter this, you should be able to select the NAT gateway from the dropdown list, to save you entering the ID letter by letter)

Hit “Save” and that’s it in the VPC interface! Give yourself a pat on the back, make yourself a coffee or something rewarding - you did good! :smile: :coffee:

Amazon EC2

Now, all that remains is to launch an instance into your new “public” and “private” subnets. Head into EC2 and click “Launch Instance”. Choose your favourite flavour OS (Linux, I hope! :wink:); I just went for Amazon Linux as I was being lazy and couldn’t be bothered finding a CentOS AMI.

When launching the instance, care needs to be taken on the “Configure Instance Details” screen. Select the correct VPC in the “Network” dropdown, select the “public” subnet and enable “Auto-assign Public IP” as we need this instance to be accessible from the internet and therefore requires a public IP to be assigned to it. Other than that, continue to launch the instance with your usual configuration. Make a note of the public IP address that get’s assigned to the instance.

After that, we need to launch another instance into the “private” subnet, which will serve the role of the application server in this example setup. Hit the “Launch Instance” button once more and again: take care on the “Configure Instance Details” screen. Here, you need to select the correct VPC in the “Network” dropdown, select the “private” subnet and do not enable “Auto-assign Public IP”. Launch the instance and make a note of the internal IP address that EC2 assigns the instance when it’s been created.

You should now have two instances running; one in the “public” subnet and another in the “private” subnet. You should be able to SSH onto the instance in the “public” instance, but not to the instance in the “private” subnet.

Testing

SSH on to the instance in the “public” subnet. As this instance is in the same VPC as the other instance which is on a “private” subnet, you should be able to SSH from the “public” instance to the “private”. Once you’re onto the “private” instance, you should then be able to access resources on the internet. For example, you can test your internet connectivity by pinging Google’s DNS servers:

ping 8.8.8.8

…and you should see replies!

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=51 time=18.795 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=51 time=19.609 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=51 time=19.644 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.795/19.349/19.644/0.392 ms

Hit Ctrl + C to cancel that ping.

Your internet connectivity on this “private” instance will be routed through the NAT Gateway that we created earlier. You can see this if you get your public IP address whilst logged into the instance:

curl icanhazip.com

…and that will return your current public IP. Note that this is different to the IP address of the publicly accessible instance that you’re connected through as this is a different internet gateway than the NAT Gateway.

Summary

So, there’s a lot to take in there! Here’s a recap of what we’ve achieved:

We created a new VPC.
We created two VPC subnets:
- One “public” where our public facing load balancer will be situated.
- Another “private” subnet, where our more vulnerable and valuable app servers will be situated.
We created a route table for the “private” subnet, that routed traffic destined for the internet through a NAT Gateway.
We created a NAT gateway that will allow our private servers to access the internet securely without exposing them to the internet.
We launched two EC2 instances:
- One publicly accessible on the “public” subnet: Demonstrates a load balancer in this example.
- One situated on the “private” subnet: Demonstrates the app server.

In the setup that we built, one can connect to the public instance directly over the internet, just like a user’s browser would be able to connect to it using HTTP directly over the internet. However, one cannot connect to the private instance, unless you are connecting from an instance which is situated in the same VPC as the private instance. In this example, the load balancer has an interface in that VPC and can therefore connect to the private instance to forward HTTP connections onto it without exposing the private instance to the internet. At the same time, both instances have full outbound internet connectivity; allowing code to be deployed to the instances, along with OS and application updates, installation and downloads.

I hope this has helped; it took me a fair bit of trial and error to work out how to do this. If this has helped, please let me know!

Careful with your Git remote config

Thu, 14 Apr 2016 00:00:00 +0000

For some time, I had the following in my ~/.gitconfig:

[remote "origin"]
    prune = true

I like keeping my branches tidy on the projects I work on and the more developers that are working on a project, the more branches are getting created, merged and deleted all the time. The above Git config means that every time you fetch, Git will prune remote branches automatically, just like it would if you used git fetch --prune. This worked great and I thought I was being clever! :bowtie:

However, recently when I built the repositories for a few of my new projects (personal and work), I noticed a problem when I came to add the remote GitHub or BitBucket (I know, I know - we have to use BitBucket at work - it’s bloody terrible!). As normal, I’d issue the following command to add the remote repo:

$ git remote add origin https://github.com/JoeNyland/test.git

But this would throw the error:

fatal: remote origin already exists.

Bugger! :angry:

Git (as always) wasn’t lying either!

I was puzzled for a while trying to work out why, on a newly created and blank repo, Git was complaining that an origin already existed. Git (as always) wasn’t lying either!

$ mkdir test
$ cd test
$ git init
$ git remote -v
origin

Whilst the above configuration conveniently sets the configuration for a remote repository to always prune remote branches when fetching, having this set implies that a remote with the name origin already exists, hence the phantom origin in the above example shows up when listing remote repositories, even though technically is does not exist.

fatal: remote origin already exists.

Googling for the error above yields many many suggestions for users to use the following:

$ git remote set-url origin https://github.com/JoeNyland/test.git

But that doesn’t really address the problem here. The real solution to that error is to remove the configuration at the top of the post from your Git config file (.git/config or ~/.gitconfig), which is just what I did and the error went away!

It’s annoying that you can’t set the default configuration for a remote that does not necessarily exist yet in your Git config, but hopefully this will be possible in a future version of Git.

Update 17/04/2015

I’ve found a solution! :tada:

Thanks to Alberto Grespan’s blog post I found that you can set the default behaviour for git fetch to prune remote branches automatically with the following command:

git config fetch.prune true

Or the following in your Git config:

[fetch]
  prune = true

I’ve added this to my ~/.gitconfig and have been using it for the past few days and it works a treat!

Moral of the story: Git is awesome!!! :metal:

A quick Rake task to tweet about a new post

Wed, 16 Mar 2016 00:00:00 +0000

I use Twitter a fair bit, so I like to fire a quick tweet out to my many (not) followers when I’ve published a new post :bird:. This task was a bit of a pain, so I thought I’d throw together a quick Rake task to do this for me.

To get this to work, you’ll need to install the twitter gem manually or, by adding it to your Gemfile.
You’ll also need to create config file that will store the API keys for the twitter gem to use to authenticate with the Twitter API. Use the following template, enter your API keys and save it as twitter-api-credentials.yaml right next to your Rakefile:

consumer_key:
consumer_secret:
access_token:
access_token_secret:

You can get your API keys by creating yourself an app at the Twitter dev site.
Add the following task to your Rakefile:

require 'twitter'

desc 'Tweet about a new post'
task :tweet_link_to_post , [:title, :url] do |task,args|

  # Connect to the Twitter API
  client = Twitter::REST::Client.new do |config|
    YAML.load_file('twitter-api-credentials.yaml').each do |config_item,value|
      config.send "#{config_item}=", value
    end
  end

  # Check that we've been provided with a title and URL
  fail 'Missing title or URL' if args[:title].nil? or args[:url].nil?

  # Confirm the message with the user to make sure they're happy with it
  puts 'This is the message that will be tweeted:'
  puts message = "I just published a blog post: “#{args['title']}” Check it out here: #{args['url']}"
  puts 'Happy with it? Enter yes or no:'
  response = STDIN.gets.chomp

  if %w{y yes}.include? response.downcase
    # User is happy, so let's tweet it!
    client.update message
    puts 'Ok, tweeted that to your followers!'
  elsif %w{n no}.include? response.downcase
    # User has changed their mind, so don't tweet
    puts 'Not tweeting, as requested.'
  else
    # Invalid response, so
    fail 'You must answer yes or no'
  end

end

When you’re ready, run the task with:

rake tweet_link_to_post[$TITLE_OF_POST,$URL_OF_POST]

… but obviously change the $TITLE_OF_POST and $URL_OF_POST parameters to those of the post that you’ve just published :smile:.