readme updates

README.md

@@ -11,11 +11,11 @@ A baseline Clojure stack that runs on any and all platforms, that is nice to dev
 ## What's here?
 
 * Leiningen based multi-module project setup with modules for each platform
-  * API via bidi with REPL control
-  * Web via clojurescript / re-frame
-  * iOS via clojure-objc / lein-objcbuild
-  * Androd via clojure-android / lein-droid
-  * Core via component
+    * API via bidi with REPL control
+    * Web via clojurescript / re-frame
+    * iOS via clojure-objc / lein-objcbuild
+    * Androd via clojure-android / lein-droid
+    * Core via component
 * Integration with Vagrant and Marathon
 * Scalable production deploment configuration
 * Development environment mirroring production environment
@@ -25,6 +25,134 @@ A baseline Clojure stack that runs on any and all platforms, that is nice to dev
 
 ## Philosophy
 
+Much influence has been drawn from Continuous Delivery, The Reactive Manifesto, and the Lean Startup. Talks such as 
+[Turning the Database Inside out]() and ___ have also had large influence.
+
+
+### Aggregate Objects
+
+After two years of clojure and data stream programming, one begins to think of all objects as aggregates of the
+events that it they have experienced. What is a key/value store? A sequence of keys and values. How can we build one?
+
+1. `(write {:key "harlan.name.first" :value "Harlan"})`, we'll use `[harlan.name.first Harlan]` for brevity.
+2. `[harlan.name.last Iverson]`
+3. `[harlan.job Hair Stylist]`
+4. `[alice.name.first Alice]`
+5. `[alice.name.last Simply]`
+6. `[alice.job Programmer]`
+
+```
+{"harlan.name.first" "Harlan"
+ "harlan.name.last" "Iverson"
+ "harlan.job" "Hair Stylist"
+ "alice.name.first" "Alice"
+ "alice.name.last" "Simply"
+ "alice.job" "Programmer"}
+```
+
+7. `[harlan.job Programmer]`
+
+```
+{"harlan.name.first" "Harlan"
+ "harlan.name.last" "Iverson"
+ "alice.name.first" "Alice"
+ "alice.name.last" "Simply"
+ "alice.job" "Programmer"
+ "harlan.job" "Programmer"}
+```
+
+The rule here is to have one slot per key that can hold a value, and keep the latest one we receive; remember, 
+these are write events. Thankfully Computer Science gave us the Dictionary data structure that we can leverage:
+
+```clojure
+(def assoc clojure.core/assoc) ; not needed, just for example. works on dictionaries, ka maps.
+
+(defn kv-store []
+  (let [store (atom {})]
+    (fn [event] (swap! store assoc (:key event) (:value event))))
+```
+
+We can image more interesting objects, like summation or moving average. I am using objects more in the logic sense than
+the OOP sense.
+
+```clojure
+(defn- inc [value] (+ 1 (or value 0)))
+
+(defn sum []
+  (let [value (atom 0)]
+    (fn [event] (swap! value inc)))
+```
+
+```clojure
+(defn- cma-pair [[i avg] value]
+  (let [new-i (+ 1 i)
+        new-avg (/ (+ avg value) new-i)]
+    [new-i new-avg]))
+
+(defn cma []
+   (let [state (atom [0 0])]
+     (fn [event] (second (swap! state update cma-pair (:value event))))))
+```
+
+
+These are foundational events. Let's try something more advanced:
+
+
+```clojure
+(defn cma []
+   (let [state (atom [0 0])]
+     (fn [event] (second (swap! state update cma-pair (:value event))))))
+```
+
+
+
+
+Stream Middleware.
+
+https://github.com/pyr/riemann-kafka
+
+Riemann + Kafka...
+
+
+http://allthingshadoop.com/2014/04/18/metrics-kafka/
+
+
+
+
+
+These aggregate objects can be composed using Clojure's inbuilt `(comp)` function.
+
+```
+;
+
+(defrecord EventsByCountry [users number-of-users]
+  (init []
+    {:users (atom (or users (kv-store)))
+     :number-of-users (atom (or number-of-users (count)))})
+  (process [forward events event]
+    (swap!  )))
+
+(let [users (kv-store)
+      number-of-users (count)]
+  (do (users event)
+      (number-of-users event)))
+```
+
+
+They can then be placed in pipelines as xforms on core.async channels.
+
+```
+;
+
+(def input (chan))
+
+(def 
+
+
+```
+
+
+
 ### Claims
 
 
@@ -36,6 +164,7 @@ A baseline Clojure stack that runs on any and all platforms, that is nice to dev
 * EDN minimally represents all fundamental data structures; the end game of text formats
 * Clojure is easy to read and understand, and is not quite LISP (it's EDN)
 * Investment in Open Source Software leads to better software product outcomes
+* Dependencies between features should be minimized, yet code sharing should be maximized and DRY
 
 
 ## Choices
@@ -48,7 +177,69 @@ The fundamental data structures representing code and data; nothing more, nothin
 * core.async channels and CSP concurrency everywhere
 * EDN everywhere, successor to JSON
 
-#### re-frame
+#### core.async
+
+Core.async provides buffered and unbuffered channels and macros to enable CSP concurrency, effectively making all arguments
+for Go apply to Clojure in addition to the tooling and library support of the JVM. Additionally there is full integration
+with transducers, allowing for composable and high performance computation (low gc pressure via stack). If we write all
+components as transducers or reducers, we can run code anywhere and wire it together with core.async channels.
+
+```clojure
+(defn assoc-profile []
+  (fn [event next]
+    (let [profile (profiles (:myapp/userid event))
+          new-event (if profile
+                        (assoc event :profile profile)
+                        event)]
+      (next new-event))))
+```
+
+```clojure
+(defn allow-language [lang]
+  (fn [event next]
+    (when (= (:myapp/lang event) lang)
+      (next event))))
+```
+
+
+```clojure
+
+(defn kv-store 
+  ([key-fn val-fn]
+     (kv-store key-fn val-fn (atom {}))
+  ([key-fn val-fn kv-atom]
+     (fn [event next]
+       (let [k (key-fn event)
+             v (val-fn event)]
+         (next (case v
+                     ::delete (swap! kv-atom disassoc k)
+                     ; if v is a list then the first will be the function and the rest the args.
+                     ; eg. 
+                     ; {:component/event :kv-store/update
+                     ;  :kv-store/key 123
+                     ;  :kv-store/update-fn '(+ (:total-price event) (or prev 0))}
+                     ::update (let [update-fn (eval (:kv-store/update-fn update-fn))]
+                                (swap! kv-atom update k update-fn event))
+                     :else (swap! kv-atom assoc k v)))))
+                     
+                     
+(def latest-value kv-store) ; alias
+
+(defn nearby [user-id other-user-id max-distance] (fn [event] (and (not= (:user-id %) user-id) (>= max-distance (distance 
+
+(defn users-nearby [users]
+  (fn [event next]
+    (let [user-id (:user-id event)
+          location (:location event)]
+      (swap! users assoc user-id location)
+      (let [nearby-users (into [] (comp (filter (fn [[k v]]
+                                                (when (distance )))])))
+
+
+```
+
+
+#### reagent
 
 Based on reagent, which using a special atom as the client side sate. Stacks a nice pattern for composable reactive
 processing of UI actions. Plays well with CQRS and Doman Driven Design.
@@ -61,7 +252,15 @@ A separation of concerns model for use on all platforms.
 We use a custom lifecycle dispatcher that uses multimethods, as suggested by the author's documentation.
 
 Actually this style [is much slower](http://insideclojure.org/2015/04/27/poly-perf/) than Protocol based dispatch, 
-but very nice to work with.
+but very nice to work with. [Or not](http://stackoverflow.com/questions/28577115/performance-of-multimethod-vs-cond-in-clojure).
+
+```
+Multimethods: 6.26 ms
+Cond-itionnal: 5.18 ms
+Protocols: 6.04 ms
+```
+
+
 
 ```clojure
 (ns my.component
@@ -91,6 +290,7 @@ but very nice to work with.
 A comparable protocol-based approach loses some of its niceness, but may win in dispatch speed. A macro could be constructed
 to use an in-memory map based dispatch (ie. constant lookup time).
 
+
 ```clojure
 (ns my.component
   (:require [clojure.tools.logging :refer [debugf infof]
@@ -115,15 +315,67 @@ to use an in-memory map based dispatch (ie. constant lookup time).
 (def component (->Reactor {::my-message my-message}))
 
 ;(react component event)
-
 ```
 
 
 ### Marathon deployment
 
+12 Factor App
+
 Deploy non-Docker packages. Nothing in particular against Docker, but I don't see the case for it with Marathon 
 and debootstrap.
 
+Marathon gives the concept of [Application Groups](https://mesosphere.github.io/marathon/docs/application-groups.html), which inject environmental variables into
+containers which con be used for discovery.
+
+
+> Applications can be nested into a n-ary tree, with groups as branches and applications as leaves. Application Groups are used to partition multiple applications into manageable sets.
+
+
+* *kafka*
+* notes service - kafka
+* web app - notes service
+* ios service - notes service
+* android service - notes service
+* ios app - *app store*
+* android app - *android market*
+
+This is the fully expanded view. Initially you may have a single service for all apps, but it is recommended
+to at-least create distinct routes for each type of view. The reason for this is to minimize dependencies. Italic
+means external / not explicitly deployed with this app group.
+
+
+```
+{"id": "notes.prod",
+ "apps": [{"id": "notes.notes-service.prod",
+           "command": "ENV=prod KAFKA=kafka:9092 java -jar notes-service.jar"},
+          {"id": "notes.web.prod",
+           "command": "ENV=prod NOTES_SERVICE=notes.notes-service.prod java -jar notes-web-standalone.jar",
+           "dependencies": ["notes.notes-service.prod"]},
+          {"id": "notes.ios.prod",
+           "command": "ENV=prod NOTES_SERVICE=notes.notes-service.prod java -jar notes-ios-standalone.jar",
+           "dependencies": ["notes.notes-service.prod"]},
+          {"id": "notes.android.prod",
+           "command": "ENV=prod NOTES_SERVICE=notes.notes-service.prod java -jar notes-android-standalone.jar",
+           "dependencies": ["notes.notes-service.prod"]}
+          ]}
+```
+
+#### Mesos DNS (service discovery)
+
+http://mesosphere.github.io/mesos-dns/
+
+Using this, DNS names have a convention based on the container ID. We could thus be implicit about DNS names of
+services, but it is better practice to pass them in explicitly at deployment/run time for separation of concerns.
+
+
+#### Backend per view
+
+This means that we have an endpoint for iOS, Android, Web. Core logic is required by each, but the view details
+may vary slightly if only in deployment schedule.
+
+
+
 ### Ansible
 
 familiar with it. yaml basaed DSL. as simple as you want it to be. downside? not edn.
@@ -133,6 +385,25 @@ familiar with it. yaml basaed DSL. as simple as you want it to be. downside? not
 APT
 debootstrap
 
+We can bootstrap our deployments down to the Kernel level if need-be, but ideally we're deploying to an ubuntu
+system that has all of our dependencies such as a JDK provisioned by Ansible or Packer.
+
+A `.deb` gives us system-level instructions for how to install a package. It may contain upstart info, or other
+ways to invoke the services it contains. Given that container systems want an entry point, that functionality
+is largely un-used. One could interpret the upstart init script to build a command to run.
+
+
+
+### Kafka
+
+Kafka is a replicated append-only log service that works in sergments of key/value messages. The key can serve as
+a primary key with compacted storage or a partition key to balance work amongst subscribers. In our case it is the
+primary data store from which our aggregate state is built.
+
+It serves as a persistent store and buffer against back-pressure and cascading failures.
+
+
+
 ## Tool suggestions
 
 IntelliJ Idea with Cursive (free licenses for OSS!)
@@ -146,3 +417,113 @@ homebrew + cask
 ## Thanks
 
 Thanks to all the authors and contributors of the projects used and literature referenced.
+
+
+
+## Different types of compnents
+
+Think of a component as a binary, and channels as standard in/out. Some binaries run on servers, others on phones, others 
+on desktop computers, etc. Each has different capabilities, a common one being POSIX. When you write a component in Clojre,
+you think in terms of the capabilities of a system. Component helps us manage these.
+
+
+```clojure
+
+(defn example-system [config-options]
+  (let [{:keys [host port]} config-options]
+    (component/system-map
+      :config-options config-options
+      :pubsub (pubsub (:pubsub-upstream config-options))
+      :ga (google-analytics (:ga-config config-options))
+      :app (component/using
+             (ios-client config-options)
+             {:database  :db
+              :scheduler :scheduler}))))
+              
+(def system (component/start (example-system {}))
+
+
+
+
+```
+
+
+
+Idea:
+
+- routing for subkey balancing -- atomic in ZK w/ zk-atom
+- browser creates atom that is materialized from upstream
+- synchronize atom into r/atom (perf of calculating update?)
+- atom over redis/memcache? -- if you need this, you might be designing wrong. kafka is master, partition is everything.
+- subscribe with a predicate, channel is a special case. #(= (:channel %) "my-channel")
+
+
+```clojure
+(defn my-thing [main-ui] [:div "regular reagent component"])
+
+(defrecord MainUI [loc]
+  c/Lifecycle
+  (start [c]
+     (r/render-component (my-thing c) loc))
+  (stop [c] 
+     (comment "cleanup")))
+
+(let [ps (pubsub ...)
+      events-upstream (subscribe ps #(= (:channel %) "my-channel")) ; a channel
+      events (pipe events (r/atom) false) ; pipe returns "to" channel
+      system-map (c/system-map {:ps ps :events events})]
+  (reset! system (c/start system-map))
+```
+
+
+```clojure
+(defn new-session [req]
+  (comment "create kafka consumer subsription, pipe updates through reactor and out to client."))
+
+(defn component-handler [req]
+  (let [input-chan (pipe (http-channel req) (new-session req) false)]))
+
+(defrecord NginxAppServer [port]
+  c/Lifecycle
+  (start [c]
+    (http-)
+  (stop [c])
+  
+  WithRingApp
+  (app [c]
+    (let [handler (comment "normal ring handler, middleware etc")]
+      handler)))
+```
+
+
+
+```clojure
+(defrecord Subscription [in out predicate])
+
+(def subscriptions (atom {})) ; this could be a zk-atom... = coordination among peers >:]
+
+(defn publish [req]
+  (kafka/publish ...))
+
+(defn subscribe [req]
+  (swap! subscriptions assoc sub-id (->Subscription ...))
+  
+(defn history [req]
+  (let [range (comment "offset, count from request")]
+    (comment "git history from requested offset))
+
+(defn unsubscribe [req])
+
+(defn update-consumers [watch-key consumers old new]
+  (let [[old-subscriptions new-subscriptions common-subs] (clojure.data/diff old new)
+        to-add (comment "find add from diff")
+        to-remote (comment "find remove from diff")]
+    (doseq [to-add]
+      )
+    (doseq [to-remove]
+      )))
+
+; keep kafka consumers in sync in a decoupled way :)
+(add-watch subscriptions :update-consumers update-consumers)
+
+```