Tuesday, July 24, 2018

Simple Bootstrapping of Clojure Web Service

Couple of things required to build a production quality Clojure web service includes picking a web server, config, logging, instrumentation. The Clojure philosophy is more like the Unix one where we choose the required libraries and compose them together to build the web service. And the base of all that is the ring specification. Here I will describe a simple way to piece things together.

For web server, aleph which is build on top of netty is superb as it is async in nature and has cool instrumentation capabilities if one requires such kind of optimisation, which by default wouldn't be much of a concern. We will use compojure for routing. Any typical web service having a handful of routes would be fast enough with compojure which serves its purpose well. As for logging using logging context, we will use logback because it works well when building an uberjar as opposed to log4j2.

As for configuration, we will keep it in properties file. Both the logging and app config will use the same properties file. If we require multiple properties file as in base properties and environment specific ones, we can run a pre-init script to combine them and pass it to the service to bootstrap. Now to the code part.

1. Create a lein project. The folder structure is similar to below.
├── README.md
├── config
│   ├── dev.properties
│   └── prod.properties
├── logs
├── project.clj
├── resources
│   ├── base.properties
│   └── logback.xml
├── scripts
│   ├── boot.properties
│   └── fusion.clj
├── src
│   └── com
│   └── qlambda
│   └── bootstrap
│   ├── constants.clj
│   ├── logger.clj
│   ├── services
│   │   └── persistance.clj
│   ├── utils.clj
│   └── web.clj
└── test
2. We will use lein-shell to execute the fusion.clj which will combine the config files. This can be written in any language and called into from lein.
;; project.clj
(defproject bootstap "0.0.1"
;; ...
:main com.qlambda.bootstrap.web
:prep-tasks [["shell" "boot" "scripts/fusion.clj"] "javac" "compile"]
:profiles {:dev {:repl-options {:init-ns com.qlambda.bootstrap.web}}
:uberjar {:aot :all
:main ^skip-aot com.qlambda.bootstrap.web
:jvm-opts ["-server" "-XX:+UnlockCommercialFeatures" "-XX:+FlightRecorder"]}}
:source-paths ["src"]
:test-paths ["test"]
:target-path "target/%s"
:jvm-opts ["-server"]
:min-lein-version "2.7.1"
:plugins [[lein-shell "0.5.0"]]
;; ..)
3. This is the boot-clj script which combines the config. The combined config will be present in config/{env}/config.properties file.
#!/usr/bin/env boot

;; Fuses base config with APP_ENV exported config.
;; Part of build pipeline.

:source-paths #{"src"}
:dependencies '[[org.clojure/clojure "1.9.0"]
[ch.qos.logback/logback-core "1.2.3"]
[ch.qos.logback/logback-classic "1.2.3"]
[org.clojure/tools.logging "0.4.0"]])

(require '[clojure.tools.logging :as log]
'[clojure.java.io :as io])

(import '[java.util Properties]
'[java.io File])

(defn get-abs-path []
(.getAbsolutePath (File. "")))

(def base-config-path "resources/base.properties")
(def config-dir (format "%s/config" (get-abs-path)))
(def env-xs ["dev" "prod"]) ; envs for which config will be generated

(defn prop-comment [env]
(format "App config (fusion generated) for %s.\nSome things are not ours to tamper with." env))

(defn unified-config [env]
(format "%s/%s/config.properties" config-dir env))

(defn store-config-properties [unified-props env]
(log/info (str "Unified config path: " (unified-config env)))
(with-open [writer (io/writer (unified-config env))]
(.store unified-props writer (prop-comment env)))
(log/info (str "Bosons generated for " env)))

(defn create-env-config-dirs [env-xs]
(doall (map (fn [env]
(let [env-dir (format "%s/%s" config-dir env)]
(when-not (.exists (File. env-dir))
(.mkdirs (File. env-dir))))) env-xs)))

(defn load-config-properties []
(log/info "Loading particles.")
(create-env-config-dirs env-xs)
(let [prop-obj (doto (Properties.) (.load (io/input-stream base-config-path)))
app-props (map (fn [env] (doto (Properties.) (.load (io/input-stream (str config-dir "/" env ".properties"))))) env-xs)
unified-props (map (fn [env-props]
(doto (Properties.)
(.putAll prop-obj)
(.putAll env-props))) app-props)]
(doall (map #(store-config-properties %1 %2) unified-props env-xs))
(log/info "Fusion process complete.")))

(defn -main [& args]
(log/info "Starting particle fusion system.")

4. The web layer which does the service initialisation.
;; web.clj
(ns com.qlambda.bootstrap.web
"Web Layer"
(:require [aleph.http :as http]
[compojure.core :as compojure :refer [GET POST defroutes]]
[ring.middleware.params :refer [wrap-params]]
[ring.middleware.reload :refer [wrap-reload]]
[ring.middleware.keyword-params :refer [wrap-keyword-params]]
[ring.middleware.multipart-params :refer [wrap-multipart-params]]
[clojure.java.io :as io]
[clojure.tools.nrepl.server :as nrepl]
[clojure.tools.namespace.repl :as repl]
[com.qlambda.bootstrap.constants :as const]
[com.qlambda.bootstrap.utils :refer [parse-int] :as utils]
[com.qlambda.bootstrap.logger :as log])
(:import [java.util Properties])

(defn test-handler
(log/info "test-handler")
{:status 200
:body "ok"})

(defroutes app-routes
(POST ["/test"] {} test-handler))

(def app
(-> app-routes

; --- REPL ---
(defn start-nrepl-server []
(deliver const/nrepl-server (nrepl/start-server :port (parse-int (:nrepl-port @const/config)))))

; --- Config ---
(defn load-config []
(let [cfg-path (first (filter #(not-empty %1) [(System/getenv "APP_CONFIG") const/default-config-path]))]
(doto (Properties.)
(.load (io/input-stream cfg-path)))))

(defn -main
"Start the service."
[& args]
(deliver const/config (utils/keywordize-properties (load-config)))
(log/info "Starting service.")
;; other initializations ..
(http/start-server #'app {:port (parse-int (:server-port @const/config))})
(log/info "Service started."))
;; constants.clj
(ns com.qlambda.bootstrap.constants
"App constants and configs"
(:require [clojure.tools.namespace.repl :as repl]))


(def config (promise)) ;keywordized properties map
(def nrepl-server (promise))
(def default-config-path "config/config.properties")
;; utils.clj
(ns com.qlambda.bootstrap.utils
"Helper functions"
(:require [aleph.http :as http]
[wharf.core :as char-trans]
[com.qlambda.bootstrap.constants :as const]
[com.qlambda.bootstrap.logger :as log]))

(defn parse-int [str-num]
(Integer/parseInt str-num)
(catch Exception ex 0)))

(defn keywordize-properties
"Converts a properties map to clojure hashmap with keyword keys"
(into {} (for [[k v] props] [(keyword (str/replace k #"\." "-")) v])))

(defn wrap-logging-context [handler]
(fn [request]
(binding [log/ctx-headers (char-trans/transform-keys char-trans/hyphen->underscore (:headers request))]
(handler request))))
5. The logger module helps to log arbitrary objects as json using logback markers as well. The custom logger is used in this example with net.logstash.logback.encoder.LogstashEncoder so that the logs can be pumped to Elasticsearch via logstash keeping the custom ELK data format.
(ns com.qlambda.bootstrap.logger
"Logging module"
(:import [net.logstash.logback.marker Markers]
[org.slf4j Logger LoggerFactory]))

(declare ^:dynamic ctx-headers)

(defn marker-append
"Marker which is used to log arbitrary objects with the given string as key to JSON"
[ctx-coll marker]
(if-let [lst (not-empty (first ctx-coll))]
(recur (rest ctx-coll) (.with marker (Markers/append (first lst) (second lst))))

(defmacro log-with-marker [level msg ctx]
`(let [logger# (LoggerFactory/getLogger ~(str *ns*))
header-mrkr# (Markers/append "headers_data" ctx-headers)]
(condp = ~level
:trace (when (.isTraceEnabled logger#) (.trace logger# (marker-append ~ctx header-mrkr#) ~msg))
:debug (when (.isDebugEnabled logger#) (.debug logger# (marker-append ~ctx header-mrkr#) ~msg))
:info (when (.isInfoEnabled logger#) (.info logger# (marker-append ~ctx header-mrkr#) ~msg))
:warn (when (.isWarnEnabled logger#) (.warn logger# (marker-append ~ctx header-mrkr#) ~msg))
:error (when (.isErrorEnabled logger#) (.error logger# (marker-append ~ctx header-mrkr#) ~msg)))))

(defmacro trace
`(log-with-marker :trace ~msg []))
([msg ctx]
`(log-with-marker :trace ~msg [~ctx])))

(defmacro debug
`(log-with-marker :debug ~msg []))
([msg ctx]
`(log-with-marker :debug ~msg [~ctx])))

(defmacro info
`(log-with-marker :info ~msg []))
([msg ctx]
`(log-with-marker :info ~msg [~ctx])))

(defmacro warn
`(log-with-marker :warn ~msg []))
([msg ctx]
`(log-with-marker :warn ~msg [~ctx])))

(defmacro error
`(log-with-marker :error ~msg []))
([msg ctx]
`(log-with-marker :error ~msg [~ctx])))
An example log generated is of the format
{"@timestamp":"2018-07-24T18:44:47.876+00:00","description":"test-handler","logger":"com.qlambda.bootstrap.web","thread":"manifold-pool-2-3","level":"INFO","level":20000,"headers_data":{"host":"localhost:8080","user_agent":"PostmanRuntime/6.4.1","content_type":"application/x-www-form-urlencoded","content_length":"48","connection":"keep-alive","accept":"*/*","accept_encoding":"gzip, deflate","cache_control":"no-cache"},"data_version":2,"type":"log","roletype":"bootstrap","category":"example","service":"bootstrap","application_version":"0.0.1","application":"bootstrap","environment":"dev"}
6. The logback config xml file follows. This uses the config from properties file specified by the APP_CONFIG.
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="${log.config.debug:-false}">
<property file="${APP_CONFIG}" />
<appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="FILE">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<appender class="ch.qos.logback.core.ConsoleAppender" name="CONSOLE">
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%d{HH:mm:ss.SSS} %green([%t]) %highlight(%level) %logger{36} - %msg%n</pattern>
<logger additivity="false" level="${log.bootstrap.level:-debug}" name="com.qlambda.bootstrap">
<appender-ref ref="FILE"/>
<appender-ref ref="CONSOLE"/>
<root level="${log.root.level:-info}">
<appender-ref ref="FILE"/>
<appender-ref ref="CONSOLE"/>
<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
7. Sample config files, which will be combined and the properties in the later ones replaces the parent properties.
# base.properties
# --- app props ---
# ...

# --- server props ---

# --- logging ---
# dev.properties
# --- app props ---
8. Couple of helper shell scripts
# run

function startServer {
echo -e "\nStarting server .."
lein do clean
export APP_ENV="dev"
export APP_CONFIG="config/dev/config.properties"
lein run

# nrepl

lein repl :connect localhost:8081
# repl

export APP_ENV="dev"
export APP_CONFIG="config/dev/config.properties" # const: will be auto-generated
lein repl
# release

function release {
echo -e "\nGenerating uberjar .."
lein do clean
export APP_ENV="dev"
export APP_CONFIG="config/dev/config.properties"
lein uberjar

The above code snippets would help in bootstapping a production grade clojure web service in no time with a great deal of flexibility. As for instrumentation, new relic does a great job. And this does not require any Mary Freaking Fancy Poppins frameworks which is more trouble than being helpful.

Thursday, July 5, 2018

macOS - The Dark Side with Mojave

Dark UI has always excited me. It brings a touch of sophistication to the products. For example, Final Cut Pro has a dark interface and so does trading terminals. And finally, with macOS Mojave, it is official that macOS includes a dark mode. And dark mode in Xcode 10 is superb. Playground is getting better for ML works as notebook. Can't wait to the official release of these.

jamf Saga Continues but Little Snitch to the Rescue

In my previous post, I was explaining on jamf and its privacy implications. jamf is very configurable. Since writing to disk is a potential problem, as the user can readily inspect, the alternate technique is to ping the command and control center every time the computer unlocks, wakes from sleep, boots, and such. It is a network call and unless one does outbound traffic monitoring, one does not know about this event happening. And there is nothing better than Little Snitch in my humble opinion to monitor such activities. This is the missing firewall for macOS which gives me peace of mind, knowing that there are no intruders on my local system. The default firewall does wildcard inbound and outbound checks, but the level of granularity is only app based. Little Snitch has got much better granularity (IP, domain, app, time based, profile based, inbound, outbound etc.). The network gateway monitoring is a story on its own, which requires a different technique.

A faster option would be to just set the domain the jamf pings in /etc/hosts to loopback on the local interface, though I am not much sure if jamf will honour that. This is if one does not want to use Little Snitch. Care must be taken because the Little Snitch database logs all the network calls. So any secure access that needs to be not logged has to be removed from its database which can safely be done from the UI. Even though, the database is encrypted and the encryption key is securely stored in the Keychain.

Using ProtonMail with Apple Mail App

ProtonMail has transparent encryption and decryption of emails leaving and entering the local system configured for Apple Mail app, using the ProtonMail Bridge. However the caveat is that only one ProtonMail user profile should be present per account under Profiles. If there are multiple ones, Apple Mail gets confused and is not able to pick one among them, taking the mailbox offline. In case you change account details in ProtonMail website, then use the latest profile and remove the older ones under System Preferences -> Profiles.