Sublime Text for Clojure

Sublime Text actually changed the text editor landscape. Before it was all TextMate which was stealing the thunder. But that is somehow a begone era. In this post we will have a sneak peak on using Sublime Text 3 for Clojure development. At this time there is no Emacs to cider kind of integration right from within Sublime Text, but it is possible to write one if required, however, connecting to nrepl from command line would just do fine. Also Clojure is not Java development where you would require to add breakpoint and step code execution as things are fairly straight forward to understand as opposed to having bazillion class files when it comes typical enterprise Java projects. However if one needs, you can resort to Eclipse with Counterclockwise plugin which works just fine.

1. Have Package Control installed.
2. In the beginning there were no parenthesis. It all started with a Big Bang, after which the amount of parenthesis started expanding. Since Clojure has the eternal brackets from the beginning of universe, we need a better bracket highlighter.
Add the following config to the Sublime user settings, so that the bracket highlighter works properly and does not conflict with the editor's own matching.

{
    "auto_match_enabled": false,
    "match_brackets": false,
    "match_brackets_angle": false,
    "match_brackets_braces": false,
    "match_brackets_content": false,
    "match_brackets_square": false,
    "match_tags": false
}

This package is pretty cool as it will show the matched brackets in the gutter.
3. To keep the sanity in own pattern matching of brackets and since most Lisp programmers are borderline hippie, to commemorate that let us use Rainbowth. Once installed add the following palette in the path ~/Library/Application Support/Sublime Text 3/Packages/User/Rainbowth.sublime-settings which is taken from Emacs's package to make the highlighting easier on eyes when used in conjunction with Solarized Dark theme.

{
  "palettes": {
    "default": ["#707183", "#7388d6", "#909183", "#709870", "#907373", "#6276ba", "#858580", "#80a880", "#887070"]
  }
}

But Rainbowth needs to update the colour scheme so we need to do some extraction of the desired scheme. Let's say we are using Solarized Dark theme, then do the following.

$ cd "/Applications/Sublime Text.app"
$ mkdir -p ~/temp/subl
# Solarized theme is inside the legacy package 
$ cp "Contents/MacOS/Packages/Color Scheme - Legacy.sublime-package" ~/temp/subl
# For some other themes which are not in the above, use
# $ cp "Contents/MacOS/Packages/Color Scheme - Default.sublime-package" ~/temp/subl
$ cd ~/temp/subl && unzip "Color Scheme - Legacy.sublime-package"
$ cp "Solarized (Dark).tmTheme" ~/Library/Application\ Support/Sublime\ Text\ 3/Packages/

4. Add Clojure.sublime-settings in the same directory with the below contents so that by default, the source uses two spaces, which is the standard as per style guide and since we use `-` to separate words instead of camel case or `_`, the word separator helps us to easily select symbols with double click. Else, doing a double click on a word will select up to the separator.

{
    "tab_size": 2,
    "word_separators": "./\\()\"':,.;<>~!@#$%^&*|+=[]{}`~?"
}

5. Install sublime-lispindent (lispindent) plugin. Using ⌘+i will indent the selected region. Helps to maintain code style.

Theming
Next we can look into few enhancements and theming to make it easy on eyes. I generally prefer dark themes as it looks cool and tends towards Solarized Dark. While generally most articles say to use a lighter background with a darker text, it depends on various other conditions like the room lighting, and of course personal preference. Refer the Dark UI section of the post for citations.

1. Install Ayu colour scheme. We will be using Ayu Mirage with Solarized Dark theme. Add the following config under user settings.

{
    "color_scheme": "Packages/Solarized (Dark).tmTheme",
    "theme": "ayu-mirage.sublime-theme",
    "ui_big_tabs": false,
    "ui_fix_tab_labels": true,
    "ui_font_size_small": true,
    "ui_font_source_code_pro": true,
    "ui_separator": true,
    "ui_wide_scrollbars": true
}

2. Additional UI settings follows. Since I got used to Monaco font from early days of OS X, I tend to stick with it instead of newer ones like Menlo.

{
    "font_face": "Monaco",
    "font_options": [
        "subpixel_antialias"
    ],
    "font_size": 16,
    "gpu_window_buffer": true,
    "highlight_modified_tabs": true,
    "scroll_past_end": true,
    "show_full_path": true,
    "translate_tabs_to_spaces": true,
    "word_wrap": false
}

Other Enhancements
1. GitGutter to see source control changes.
2. SideBarEnhancements plugin which adds more options to the side bar right click menu.
3. EDN for Clojure EDN syntax highlighting.

Once done, the editor will look elegant as in the screenshot.



Dark UI

[1] Negative contrast computer displays (dark letters on a light background) generally provide a more legible image than positive contrast displays (light letters on a dark background). Either black characters on a white background or white characters on a black background have been found to be more visible than green, yellow-orange, blue or red characters.²⁰

[2] The greatest difference among groups is on the black & white pair: while the majority of people without dyslexia prefer it (32.67%), only 13.64% of the participants with dyslexia chose black text on white background.
Participants without dyslexia tend to prefer color pairs with a higher color and brightness contrast while people with dyslexia read faster when color pairs have lower contrasts. For instance, the color pair which was the fastest to read by the participants with dyslexia was black & creme (mean of 0.214 for the fixation duration) while black text over yellow background presents the largest fixation duration mean (0.239 seconds). Although the pair off-black & off-white is the one recommended in Web design for dyslexics (Bradford, 2011), none of the users with dyslexia selected it. This pair presents the highest fixation duration mean for the participants without dyslexia (0.193 seconds).

[3] Maureen did mention two situations when a dark background makes sense.

However, if you are using a display in a dark environment, it’s better to use a dark background as it lets you keep your eyes dark adapted. That’s why controls for airplanes and GPS units for cars switch to a dark background at night. Usually, however, the results don’t look like symbols and text floating in the darkness of space…there’s still a sense of there being a dark surface to ground the view. So the concerns above are somewhat mitigated.

I’ve seen recommendations that white on black is better for aging eyes, and for people with low vision because it reduces the amount of light scattering and distorting the image.

[4] Chief among the reasons many of you have expressed for preferring dark backgrounds is the reduced strain placed on the eyes when staring at the screen for many hours. Many developers state that light text on a dark background is easier to read over longer periods of time than dark text on a light background.

Integration Tests with pytest

pytest is a unit testing library and coupled with requests package can be used for writing integration tests. With integration tests, couple of things we would need is a way to distinguish between environments like local, staging, production, different datacenters and run tests in all or some specific tests per env and such.

1. We need a way to pass env during test runs. We will use pytest-variables for that which makes it easy to get variables passed in from command line available to the tests. Another way is to write conftest. So our requirements.txt file will now have pytest-variables[hjson]==1.5.1 apart from other packages.
2. We will name our pytest file starting which starts with test as something like test_api.py
3. Now on writing the tests. I would recommend that the python code be modular and well organised instead of spaghetti coding. Read Modular Programming with Python by Erik Westra if you are interested. We could separate parts into different file or have things organised under class and keep it under the same file. The point is separation of concerns, readability and maintainability (especially by other programmers). A thing with OOP is that go for composition as opposed to inheritance when possible.
4. We will go with how to write pytests in an object oriented fashion rather than just writing test_ functions. Let's call the below module as test_api.py

import pytest
import logging
import requests

__version__ = "0.0.42"  # corresponds to app version
logging.basicConfig(level=logging.DEBUG)
__timeout = 90  # seconds

def get_headers(props):
    """Return header map."""
    # ...

def http_post(params, data, headers=None):
    """Send an HTTP POST request."""
    return requests.post(params['server'] + params['endpoint'], data=data, headers=headers)

class FooAPI:
    """Sample Foo API helper class.""""

    def __init__:
        pass

@pytest.mark.timeout(__timeout)
class TestFooAPI:
    """Test Foo API service."""

    pytestmark = [pytest.mark.local, pytest.mark.ny1, pytest.mark.ams1, pytest.mark.ny1_green, pytest.mark.stg1]

    @pytest.fixture()
    def foo(self):
        """Return Foo object."""
        return FooAPI()

    def gen_name():
        """Standalone function that not pytest functions."""
        pass

    def test_add_user(self, variables, foo):
        """Test adding of user."""
        log = logging.getLogger('test_add_user')
        resp = foo.create_request(variables)
        headers = get_headers(variables)
        http_resp = http_post(variables, data, headers)
        assert http_resp.status_code == 200

    @pytest.mark.ams1_beta
    def test_beta_endpoint(self, variables, foo):
        """Test beta endpoint."""
        log = logging.getLogger("test_beta_endpoint")
        data = {'users': base64.b64encode('something'), 'bar': 'baz'}
        start = time.time()
        headers = get_headers(variables)
        log.info("\nheaders: %s", headers)
        http_resp = http_post(variables, data, headers)
        rtt = time.time() - start
        log.debug("\nrtt: %s\n", rtt)
        body = json.loads(http_resp.content)
        assert body['status'] is True
        assert body['code'] == 0
        assert http_resp.status_code == 200, 'Expecting 200 OK status code.'

class TestBarAPI:
    # ....

Any class that starts with Test and any function that starts with test will be executed by pytest by default. However these behaviour can be configured. Fixtures are variables that are passed to each test as argument before it runs. There are various pytest markers. If whole pytest functions within a pytest class needs to executed by many different environments, we can pass specify them as pytestmark = [pytest.mark.en1, pytest.mark.env2, ...]. The env1, env2 correspond to env config file name. And if we need to execute specific tests against specific env then, exclude them from the above list and annotate it above that particular function as @pytest.mark.ams1_beta, which means, if the env passed via command line is ams1_beta, only then this test will execute and since the env is not specified common in the pytestmark list, other tests will be skipped. Example usecase is running sanity tests (a subset of tests) against productions at a specific interval, where we do not have to run the full testsuite. The variables argument contains the config values defined in the config json which will be available to the test methods as a python dictionary.
5. The configuration files are under config directory. Example local.json is below. The name of the file corresponds to the env which we will be passing to the pytest via command line.

{
    "server": "http://localhost:8080",
    "endpoint": "/foo/test/",
    "key": "local_key.pem",
    "cert": "local_cert.pem",
    "env": "local",
    "json": true
}

6. Tying it all together using shell script. It is better to use virtualenv. So below script does a bit of housekeeping before running the tests.

#!/bin/bash

# Integration test with pytest.

TEST_DIR=test_api
VENV_DIR=venv
CONFIG_DIR=config

function activateVirtualEnv {
    echo "Activating venv .."
    if ! hash virtualenv 2>/dev/null; then
        echo "virtualenv not found. Installing .."
        pip install virtualenv
    fi
    virtualenv $VENV_DIR
    source $VENV_DIR/bin/activate
}

function installDeps {
    echo "Installing dependencies .."
    pip install -r ../requirements.txt
}

function prepareForTest {
    echo "Running integration test .."
    activateVirtualEnv
    installDeps
}

function deactivateVirtualEnv {
    echo "Deactivating venv .."
    deactivate
}

function displayhelp {
    echo "Usage: $0 options"
    echo "where options include:"
    echo "  local    run test in local env"
    echo "  ams1     run test in Amsterdam production datacenter 1"
    echo "  stg1     run test in staging enviroment 1"
}

# $1 result xml file name
# $2 env variable name
function run {
    echo "Running test against $2"
    # -n xdist flag
    # -s capture output
    # -v verbose
    # --variables The config file
    # -m marker
    py.test -n 4 -s -v --junitxml="$1" --variables "$CONFIG_DIR/$2.json" -m "$2" test_api.py
}

# $1 results.xml
# $2 env : local, rqa2, etc. corresponds to the filename in config
# To run against all machines in staging and production, use env as stg and prod respectively
# For integration test
# ./runtests.sh results.xml local
function main {
    result="$1"
    env="$2"
    echo "result: $result"
    echo "env: $env"
    echo "config: $CONFIG_DIR"

    cd $TEST_DIR
    prepareForTest
    if [[ $env == "stg" ]]; then
        run stg1.xml stg1
        run stg2.xml stg2
        python merge_junit_results.py stg1.xml stg2.xml > $result
    elif [[ $env == "prod" ]]; then
        run ny1.xml ny1
        run ams1.xml ams1
        python merge_junit_results.py ny1.xml ams2.xml > $result
    else
        # individual
        run $result $env
    fi
    return_code=$?
    deactivateVirtualEnv
}

main $@

exit $return_code

The merge_junit_results.py can be obtained from the cgoldber gist.
To run the test we run the command ./runtests.sh ${result}.xml ${env-config-file-name}, which will check for virtualenv and such and run the test. To run the tests in parallel use xdist.
7. Sample reqirements.text file would look like

pytest==3.0.7
pytest-variables[hjson]==1.5.1
pytest-xdist==1.15.0
requests==2.13.0
pytest-timeout==1.2.0

8. To integrate the results with jenkins, it needs to know the final result of the testsuite which will be present in the results.xml file. To parse that use the code below.

#!/bin/python

"""Extract test result failure, error values for Jenkins job."""

import xml.etree.ElementTree as ET
import sys


def parse_result(file):
    """Parse the given results.xml file."""
    root = ET.parse(file).getroot()
    if root.attrib["failures"] == "0" and root.attrib["errors"] == "0":
        print "ALLTESTSUCCESS"
    else:
        print "ERROR"
        sys.exit(1)


if __name__ == '__main__':
    print "junit filename: ", sys.argv[1]
    parse_result(sys.argv[1])

Now we have a full blown integration testsuite which can be integrated with CI/CD pipeline.

For clarity, sample folder structure will look like below.

test
├── requirements.txt
├── runtests.sh
└── test_api
    ├── __init__.py
    ├── local.xml
    ├── config
    │   ├── ams1.json
    │   ├── ams1_beta.json
    │   ├── local.json
    │   ├── local_beta.json
    │   ├── ny1.json
    │   ├── ny1_beta.json
    │   ├── ny1_green.json
    │   └── vm.json
    ├── merge_junit_results.py
    ├── parse_xml.py
    ├── result.xml
    ├── test_api.py
    └── venv

Passcode Lock iPhone even with TouchID Enabled

I found a way to force lock iPhone that has TouchID enabled. If done as below, then the phone requires passcode to unlock even when TouchID is configured to unlock the device.
1. Enable AssistiveTouch from Accessibility.
2. And under AssistiveTouch->Custom Actions, say for Long Press set it to SOS.

Now when we need to passphrase lock, long press the assistive touch button, which will dial the emergency SOS number. Cancel the call immediately and the phone gets locked. The only problem is that it makes the emergency sound when dialling.

Or second option is to press the unlock button and tap the wrong finger five times and the phone will lock itself. Whichever is easy works. Now how would the second option work with FaceID?

Erlang is Famous as it appeared on TV series

Did you know that Erlang is famous as in Hollywood famous, and you had not heard about it? Turns out a chat conversation in BlindSpot TV series 1 episode 6 says "Sybok: how quickly can you learn erlang?". And it indirectly implies that the TRAKZER app was written in Erlang. Anyway that is just a story. But yeah, how quickly can you learn Erlang dear?



But unfortunately, there is a twist to the story (spoiler alert). The girl (Ana Montes) at a later point in time uses python to break into the FBI system and then switches to perl. Which suggests that the above assumption is likely false or may be due to the lack of time and to give a clue to Patterson she used it. And the interesting part is the girl writes well documented code that follows PEP-257 at gunpoint. Now that is what I call software craftsmanship.

A Song by Siri

I said something, Siri heard something else and in the end she sang a song. Not bad I would say.

Solitude by Jehnny Beth

These are the words from the video Start Making Sense by Jehnny Beth on Beats 1. I like these words, the poetry.

I'm close to the burn out,
doing nothing is all I dream about.
Friendships, career, phone,
even when we're together we're all alone.
I wish I could freeze time,
make this morning last longer.
Everything I do on my own,
I remember better.
Songs to enjoy my solitude.