Getting Wireshark to work on OS X Yosemite

Half a month back I upgraded two of my Macs to Yosemite. One was my relatively new MacBook Air at home, which had Mavericks on it. The other was my slightly old MacBook Pro at work, which had Mountain Lion. To my relief, the upgrade went briskly on both systems. (Here’s a little secret: I used this guy’s excellent advice to make sure the upgrade did not take too long to finish.) Not only that, to my surprise, Yosemite improved the OS X experience–both UI and performance wise–greatly. I was afraid putting Yosemite on my MacBook Pro, in particular, might slow it down, forcing me to clean the system and attempt a fresh install. I took backups of course, as should you, but cleaning up an entire system and setting it from scratch isn’t a happy thought. As a rule, I prefer not to do upgrades for major OS X (or iOS) releases. In my experience, a clean install almost always is the better option. Upgrades across major versions of the OS are risky to do. They also additionally–if they do succeed–run the risk of slowing the OS or parts of it down afterwards because of corrupted configurations and whatever mess that was created during the upgrade. The more data you have on your system that you want to upgrade, the greater the risk of a failed or botched upgrade.

Yosemite proved otherwise. For me, at least. I have friends who, unfortunately, have reported issues after upgrading to Yosemite, but they’ve all had older Macs than I do. I can only speak for myself.

Some things did break, though. Like MacVim, which I love, but which I won’t talk about right now. On my work MacBook Pro, I had Wireshark installed that I was using from time to time to dissect network traffic. It’s a great tool. On Yosemite, it stopped working. I found suggestions from strangers on the Internet about re-installing Wireshark, dejected responses from people who did only to find it didn’t make a difference. I then came across a consensus on reinstalling X11/XQuartz instead. People shouted that it worked.

Before I went ahead and reinstalled X11/XQuartz on Yosemite, I fell upon a small gem which explained why X11/XQuartz needed to be reinstalled and how reinstalling it could be avoided. It said that Wireshark was expecting X11/XQuartz to be inside /usr when in fact in Yosemite it was now under /opt. A simple solution was to create a symbolic link inside /usr with the following harmless command on Terminal:

sudo ln -s /opt/X11 /usr/X11

Sure enough, that did cause Wireshark to start. But it took an awful lot of time to show up. X11 apps are slow and crappy in terms of responsiveness on Macs, but they don’t take that long to load. When it did show up on the screen, it failed to detect any interfaces. Now that seemed rather odd. I looked at the system logs through Console.app and figured out that it was a rather silly permission issue. The easiest but not the recommended solution was to run Wireshark as root:

sudo wireshark

I needed to get real work done and couldn’t afford to spend any more time than I had trying to figure out a better way to run Wireshark, so I went ahead with running it as root. But you shouldn’t. As it happens, there are ways to a better solution that involve setting up Wireshark with privilege separation. The Wireshark wiki has an article about it. There is a section near the bottom about OS X which does not read very positively. But there’s a solution there. If you’re finicky about running apps with root privileges–and you should be–, you should go through it. I need to play with it. When I have it all pieced together, I will write again.

Enjoy dissecting packets and analysing network traces!

When pushing a big repo over Git to Assembla fails

At work, Assembla is used extensively, not only to track tasks but also to store DSCM repositories, such as Git. I don’t like Assembla for several reasons, and I don’t like hosting Git repositories on Assembla even more.

Some months back I had to initialise and push to Assembla a Git repository that was almost 800MB big. Like any Git hosting provider, Assembla supports both the Git (SSH) and the HTTPS protocol for interacting with repositories. While I always prefer Git over SSH, there are times when you don’t have your SSH key around or don’t want to use it. In those cases, I simply use the HTTPS protocol, which is rather convenient. However, while pushing this particularly big repository over HTTPS to Assembla, I encountered the familiar “The remote end hung up unexpectedly” error message from Git. My initial thoughts were to blame a flaky network for the error. However, no matter what I tried and how many times, I kept getting that dreaded error.

I went around Assembla looking for any help, but couldn’t find anything that was really relevant to my problem. Luckily, however, I did find a knowledge base article inside Altassian’s documentation, which described the exact same problem. You may read it up here:

If you would rather read a quick rundown, then carry on.

When Git encounters packed objects greater than 1MB in size, it uses “Transfer-Encoding: chunked” in POST requests. Not every web server handles transfer encoding by default, particularly not nginx, unless it is set up to do so. Assembla by co-incidence uses nginx and apparently, the nginx configuration they have to handle Git over HTTPS isn’t set up to use transfer encoding properly. That explained why in my case, where the repository was big and therefore so were the packed objects, Assembla’s Git+HTTPS protocol couldn’t handle the push.

What was the workaround?

I could use Git over SSH and avoid the problem altogether.

Or, as the Altassian article pointed out, the post buffer size for Git could be changed to match the size of the repository. When I changed the post buffer size to 800MB with:

git config --global http.postBuffer 800000000

the next push, while it took its sweet time, worked. :)

Why take the Ice Bucket Challenge?

I first heard about the Ice Bucket Challenge when I briefly read a post shared on Facebook about Bill Gates taking on the challenge. I then read about it again where Elon Musk was soliciting help of his children to undertake the same challenge. To me, it seemed a pointless little thing. What I didn’t know was that the challenge had a much deeper meaning, which, sadly, many of the people taking the challenge neglected to mention. You could say that my ignorance is mostly to blame for it, and nothing else, and I’d partly agree with you. However, I talked to a number of people who knew about the challenge but had no clue what it was about. It’s a real shame that despite the challenge taking the Internet by storm, very few places mentioning the challenge actually took the time to explain or even mention the real deal behind the challenge.

The Ice Bucket Challenge, in and of itself, has no real significance. It serves only two purposes:

  • To raise awareness about a life-threatening disease called ALS;
  • To encourage people to take on the challenge, motivate others to follow suit, and donate to ALSA, a non-profit organisation dedicated to fighting the ALS disease.

I thought I would write a little and talk about what ALS really is.

ALS stands for “Amyotrophic Lateral Sclerosi”. It is also known as the “Lou Gehrig’s Disease”. It is a disease that affects nerve cells in the brain and the spinal cord. The human body comprises of many different kind of nerves. The nerves that provide voluntary movement of muscles as well as muscle power are called “motor neurons”. When you wish to move your limbs, motor neurons are sent from your brain to your spinal cord, and from the spinal cord to your muscles so that your muscles may move to your whims. ALS affects the generation and nourishment of those motor neurons. Under ALS, the motor neurons progressively degenerate, and eventually die. When the motor neurons die, your brain loses the ability to initiate and control any kind of voluntary muscle movement. Eventually, the person affected with ALS runs a high risk of becoming completely paralysed, leading to their death.

ALS is a life-threatening disease. It has no treatments. There are no cures. There are no ways to halt or reverse the progress of the disease. However, there is one drug which has been proven to slow down the progress of ALS, even if moderately.

The Ice Bucket Challenge is there to raise awareness about the disease. I will probably never take the challenge, because it amounts to nothing, except to serve as a lark to boast about among friends, unless it succeeds in bringing awareness about ALS to the forefront. If you wish to donate, here’s how.

Sorting by IP on nth field with sort on Linux

I found a colleague at work struggling with sorting some data in Excel on Windows (ugh). He wasn’t getting anywhere. The data was in the following format (note that I’ve obfuscated the IP addresses):

user279 Actual_IP   192.000.000.10
user243 Actual_IP   192.000.000.103
user294 Actual_IP   192.000.000.11
user316 Actual_IP   192.000.000.112
user291 Actual_IP   192.000.000.115
user277 Actual_IP   192.000.000.12
user294 Actual_IP   192.000.000.121
user273 Actual_IP   192.000.000.13
user300 Actual_IP   192.000.000.130
user285 Actual_IP   192.000.000.135
user263 Actual_IP   192.000.000.138
user279 Actual_IP   192.000.000.14
user279 Actual_IP   192.000.000.15
user287 Actual_IP   192.000.000.16
user244 Actual_IP   192.000.000.165
user216 Actual_IP   192.000.000.17
user272 Actual_IP   192.000.000.171
user259 Actual_IP   192.000.000.179
user292 Actual_IP   192.000.000.18
user275 Actual_IP   192.000.000.180
user295 Actual_IP   192.000.000.19

He wanted to sort by IP addresses on the third column. Being a crazy Unix/Linux CLI guy, I instantly thought about how I would do it with Linux. First of all, I wanted to figure out a way to sort based on data in the third column only. The man page for sort(1) revealed the -k switch. The -k switch defines the key to use for sorting. The key refers to a part of the string on a line on which sort runs. By default, the entire line is used as a key, causing sort of sort by the entire line. It works in conjunction with the -t switch which defines the field separator character. By default, the field separator character is a whitespace character (or any number of continuous whitespaces). Because the data I had was separated by spaces, I had no need to tweak it with the -t switch. With the following, I told sort to sort on the third key:

$ sort -k3,3 data.csv  

user279 Actual_IP   192.000.000.10
user243 Actual_IP   192.000.000.103
user294 Actual_IP   192.000.000.11
user316 Actual_IP   192.000.000.112
user291 Actual_IP   192.000.000.115
user277 Actual_IP   192.000.000.12
user294 Actual_IP   192.000.000.121
user273 Actual_IP   192.000.000.13
user300 Actual_IP   192.000.000.130
user285 Actual_IP   192.000.000.135
user263 Actual_IP   192.000.000.138
user279 Actual_IP   192.000.000.14
user279 Actual_IP   192.000.000.15
user287 Actual_IP   192.000.000.16
user244 Actual_IP   192.000.000.165
user216 Actual_IP   192.000.000.17
user272 Actual_IP   192.000.000.171
user259 Actual_IP   192.000.000.179
user292 Actual_IP   192.000.000.18
user275 Actual_IP   192.000.000.180
user295 Actual_IP   192.000.000.19

The 3,3 told sort to start the key at the third field and ending at the third field. Since there are no fields after the third field in my data set, using -k3 would’ve achieved the same results. However, after comparing the data-set with the result of the sort command, I found no changes. After analysing the result for a while, I finally understood what was going on. I looked closely at the man page for sort again, particularly at the available sorting operations. What caught my attention was the -V switch, or in long form --version-sort. It sorted based on version numbers in the key. Looking closely at the IP address column, I realised the IP addresses looked very much like extended version numbers. Running with the -V switch gave me this:

$ sort -k3,3 -V data.csv

user279 Actual_IP   192.000.000.10
user294 Actual_IP   192.000.000.11
user277 Actual_IP   192.000.000.12
user273 Actual_IP   192.000.000.13
user279 Actual_IP   192.000.000.14
user279 Actual_IP   192.000.000.15
user287 Actual_IP   192.000.000.16
user216 Actual_IP   192.000.000.17
user292 Actual_IP   192.000.000.18
user295 Actual_IP   192.000.000.19
user243 Actual_IP   192.000.000.103
user316 Actual_IP   192.000.000.112
user291 Actual_IP   192.000.000.115
user294 Actual_IP   192.000.000.121
user300 Actual_IP   192.000.000.130
user285 Actual_IP   192.000.000.135
user263 Actual_IP   192.000.000.138
user244 Actual_IP   192.000.000.165
user272 Actual_IP   192.000.000.171
user259 Actual_IP   192.000.000.179
user275 Actual_IP   192.000.000.180

Voila.

The Python netifaces module

Three years ago I wrote about a couple of methods for obtaining public interface IPs on a system using Python. Last week I had the need again to find IP addresses of all public interfaces on a given server using Python. Accidentally, I came across the netifaces Python module. It is succinct and provides a clean interface. Getting addresses of network interfaces in a portable manner is an extremely difficult task, but this library gets most of the job done. All one needs to do is pip install netifaces and start using it.

As a bonus, I have a pice of code to extract a list of IPs from a system’s ethX interfaces:

import netifaces

def get_public_ip_list():
    interfaces = netifaces.interfaces()
    public_ip_list = []
    for interface in interfaces:
        # Only keep ethX interfaces.
        if not interface.startswith(“eth”):
            continue
        # Discard interfaces that are up but without any IPs.
        addrs = netifaces.ifaddresses(interface).get(netifaces.AF_INET)
        if not addrs:
            continue

        ips = [addr.get(“addr”) for addr in addrs]
        try:
            public_ip_list.append(ips[0])
        except IndexError:
            pass

    return public_ip_list

How to silence nodes on Sensu

Sensu is an open source monitoring framework, written purely in Ruby, that we use heavily at Cloudways to monitor not only the servers of our customers but our own as well. Sensu is very extensive. If combined with Graphite, the pair can be used to cover a wide variety of monitoring and reporting/graphing scenarios.

While Sensu has a helpful API, what’s not very helpful is its documentation. For a lack of a better word, its documentation is sparse. While the v0.12 of its documentation attempts to fill in the gaps, it continues to fall short.

Recently, a need arose which required Sensu clients, which are servers that are running the Sensu agent, to be silenced when the servers on which they were running were stopped. Looking through the clients API, I could only find a way to delete a client, which theoretically could work, only, there wasn’t an API to re-register a client again. The documentation vaguely said that the clients registered themselves the first time they came alive.

The Sensu CLI, which isn’t an official part of Sensu, is a wonderful command line Sensu client. This client, interestingly, provides a means of silencing a client and un-silencing it. Playing with its source, I was finally able to figure out how it was doing it.

The way Senus makes possible silencing and un-silencing of clients is through the Stashes API. The important bit is the format of the path parameter. If it takes the form of /silence/{{ client_name }}, it will silence the client identified by client_name. Here’s a Python snippet to make it clear how to use the Stashes API to that effect:

#!/usr/bin/env python

import time
import json
import requests

client = “YOUR_CLIENT_NAME”

url = “http://localhost:4567/stashes/silence/%s” % client

def silence():
    payload = {
        “timestamp”: time.time()
    }

    r = requests.post(url, data=json.dumps(payload))
    print r.status_code
    print r.text

def unsilence():
    r = requests.delete(url)
    print r.status_code
    print r.text

The code should be self-explanatory. The silence() method makes a POST request to the Stashes API. The timestamp value identifies the current time in epoch. The unsilence method, in contrast, makes a DELETE request to the same Stashes API.

Note that by default the silence operation will silence the given client indefinitely. If you wish to silence it for a defined period, an expires parameter can be provided as part of the payload.

You must learn to touch type

If you love typing, you have got to read The Joy of Typing. It is everything I have ever wanted to say on the topic of learning to touch type.

I have major beef with people–especially those in professions which involve working on a computer, particularly systems engineers and development folks–who don’t know how to touch type. It infuriates me to see people hunt-and-peck type when they could have just as easily learned to touch type.

I learned touch typing many moons ago, when I was a shy O’Levels student. I had taken computers as my major in O’Levels. One day I discovered a DOS-based touch typing tutor application. I fell in love with it. I wish I could remember its name so that I could make an effort to find it. I began religiously, and very patiently–for learning to touch type is an exercise in patience as much as anything else–to learn typing properly. To your surprise, it didn’t take me long to learn typing properly–if I am forced to remember, I would say it only took me a better part of twenty or less hours to get in the right direction. Afterwards, it was only a question of how much I practised.

Those twenty hours changed the way I looked at and used computers for the rest of my life.

I am not being melodramatic. I am about as dead serious as a door nail nailed to a door can be. The moment I learned touch typing, my experience of computers and, particularly, the Internet transformed completely. And not for the bad. It felt great. Later on as I would take on a professional career in network and systems administration and software development, I would only find myself reaping the rewards of the seed I planted years before and nurtured.

Having been in the IT industry under a decade, I have found myself at countless–yes, countless–times making the following observation: Why do so many people–including systems administrators, engineers, developers–simply do not know how to touch type. It has been a frequently repeating pattern I have observed in my professional life. I’m not making an understatement when I say that watching developers and systems people, in particular, hunt-and-peck type miserably is akin to having your eyes impaled with burning needles. I simply cannot fathom why those people did not just learn touch typing.

Computers are a way to consume information as much as they are a way to express yourself. In a world where things are moving as fast as a faulty bullet train, it is extremely beneficial to be able to express yourself freely. I do not mean freely in the sense of freedom of expression. We have that aplenty. When I speak of being able to express yourself freely, I mean in terms of the mode of expression. Despite whatever advances we have made in the field of speech recognition and synthesis, keyboards continue to remain the most used forms of providing input to computers. When you can type as fast as your brain can coherently think, you suddenly begin to understand the power available at your behest. When you know how to touch type, replying to emails, writing documentation, typing code, they all become exercises that you enjoy. They no longer feel strenuous activities. If you no longer fumble and struggle with typing, you spend much more time in expressing yourself even more clearly and lucidly. You do not cringe away at the thought of writing a detailed reply to an email, or documenting something, or even writing articles and blogs. All in all, you have nothing to lose, and everything to gain, from learning to touch type. If you’re reading this and you don’t know how to touch type, do yourself a little favour! You will only thank yourself later.

As the author suggests in the article I mentioned at the beginning, I too firmly believe that colleges and universities should force students to learn touch typing. Period. The academia would do these young people a great favour as well as the industries they go on to become a part of.

keybase.io/ayaz

I’m finally on keybase.io. I also have four invites available. If you are looking for one, please send me your GPG/PGP public key ID as well as your email address, and I’ll send an invite your way.

keybase.io is cool. While still in alpha release and open to only private testers, keybase.io aims to become a public directory of publicly audit-able public GPG/PGP keys. Not only can you put up your public GPG key there, but you can also associate it with a username, and link it with your social personas online, such as, for example, Twitter, GitHub, your blog or personal website.

It has a very cool web interface, which allows you, among other things, to sign, verify, encrypt and decrypt messages sent to you or other keybase.io users. For the ClI-savvy folks like myself, it also has a command-line tool, written in node, which supports a plethora of operations with a very simple to use interface. If you frequently use GPG/PGP, you should definitely give keybase.io a go.

My public key on keybase.io is here: keybase.io/ayaz/key.asc

The state of certificate revocation in Chrome

In the wake of the OpenSSL Heartbleed bug, revocation of potentially affected certificates became an immediate necessity. There was a great risk of malicious attackers who, having stolen SSL keys, could likely set up MITM attacks using compromised certificates and keys. It became the responsibility of the owners of compromised SSL keys and certificates to revoke the certificates and keys and replace them with new ones. What most people didn’t know was that it also became the responsibility of clients and applications that exchanged and verified those SSL certificates to become aware of revoked certificates. Browsers, above all, are the set of most common applications that allow users to interact with websites and services secured with SSL. It became the responsibility of browsers to know which certificates were revoked and to alert their users when they encountered a fradulent certificate.

Chrome being one of the most popular browsers in the wild, I came across a setting in it which allowed for checking of certificate revocation. Before I could use that setting for long, I came across a number of articles which for one reason or another did not advocate the use of that feature. Back then, I didn’t quite understand why using that feature in Chrome would be a harmful thing.

Yesterday I came across an extremely long dissection of the way Chrome handles certificate revocation. It is exhaustively long, yes, but it is a harrowingly shocking read. It explains in breathtaking detail how and why Chrome’s certificate revocation is not only effective, but it can also prove to be harmful for end-users.

We know that Chrome’s CRLSet certificate revocation system is completely broken. It cannot, and should not, be relied upon to protect the security interests of its users.

If you are concerned about the security and privacy of your existence online, as I think everyone should be, I strongly recommend you hang through till the end of that article.

Pearls before breakfast

“The violin is an instrument that is said to be much like the human voice, and in this musician’s masterly hands, it sobbed and laughed and sang — ecstatic, sorrowful, importuning, adoring, flirtatious, castigating, playful, romancing, merry, triumphal, sumptuous.”

What a long but deeply moving article. It took me the better part of several hours to read it, but I am so glad I did. Even if you do not appreciate classical music (I cannot imagine why one would not, though), you will still be moved by the breathtaking beauty of this article.