Update documentation

Author: killerwhile

README.md

@@ -1,31 +1,32 @@
 Kafka Partitions Assignment Optimizer
 ====
 
-If you have more than 4 brokers spread on several top-of-rack switches (_TOR_),
-you might be interested in balancing replicas and leaders properly to
-survive to a switch failure and to avoid bottlenecks.
+If you have more than 4 brokers spread on several top-of-rack switches (_TOR_)
+or availability zones (_AZ_), you might be interested in balancing replicas
+and leaders properly to survive to a switch failure and to avoid bottlenecks.
 
-On addition to that, when you're re-assigning replicas because of server failure,
+In addition to that, when you're re-assigning replicas because of broker failure,
 or changing the topology (server(s) addition) or the replication factor,
-you might be interested in minimizing the number of partitions to move.
+you might be interested in minimizing the number of partitions to move
+to avoid killing your network.
 
-For this latter, the `kafka-reassign-partitions.sh` utility is not doing a perfect
-job at minimizing the number of replicas moves.
+For this latter, the `kafka-reassign-partitions.sh` utility provided with Kafka
+is not doing a perfect job at minimizing the number of replicas moves.
 
 To give a concrete example, adding or removing a server from the cluster is
 generating lots of replica moves (i.e. network traffic) that might impact the
-overall cluster availability.
+overall cluster performance.
 
-Also, if you running a version of Kafka which does not include
-[KIP-36 (rack aware replica assignment)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment))
+Last but not least, if you're running a version of Kafka which does not include
+[KIP-36 (rack aware replica assignment)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment)
 you don't have any knowledge about the network topology in the
 assignment algorithm.
 
 ## Demonstration: `kafka-reassign-partitions.sh` under-efficiency
 
-Lets assume we have a cluster with 20 brokers, named 0-19, spread across 2 switches.
-Brokers with odd numbers are all on the same _TOR_ `tor1`,
-brokers with even numbers are wired to `tor2`.
+Lets assume we have a cluster with 20 brokers, named 0-19, spread across 2 AZ.
+Brokers with odd numbers are all on the same _AZ_ `b`,
+brokers with even numbers are wired to `a`.
 
 We have a topic `x.y.z.t` with 10 partitions and a replication factor of 2.
 
@@ -84,7 +85,7 @@ That's rather unfortunate, since computing the diff manually,
 we could simply change the assignment of partition `1`, like:
 
 ```
-    {"topic":"x.y.z.t","partition":1,"replicas":[8,1]},
+{"topic":"x.y.z.t","partition":1,"replicas":[8,1]},
 ```
 
 All the other moves are not required.
@@ -103,13 +104,16 @@ of writing.
 
 
 # Replica assignment as a constraint satisfaction problem
- 
+
 If you think out of the box, replicas assignments looks like an 
-[optimization function](https://en.wikipedia.org/wiki/Mathematical_optimization)
-under specific constraints, or a
-[constraint satisfaction problem](https://en.wikipedia.org/wiki/Constraint_satisfaction_problem)
+[constraint satisfaction problem](https://en.wikipedia.org/wiki/Constraint_satisfaction_problem).
+
 For instance, "no two replicas of the same partition assigned to the same broker" is one of
-these constraints.
+these constraints which could be expressed as an equation, opening the door
+to [mathematical optimization](https://en.wikipedia.org/wiki/Mathematical_optimization)
+to find the optimum.
+
+## Minimize the number of replicas to move
 
 To minimize the move of replicas, the idea is to assign more weight (i.e. more value)
 to existing assignments, so that the linear optimization will try to preserve
@@ -124,7 +128,7 @@ would be expressed as
 
 ![Constraint example](images/constraint1.png)
 
-Now you got the trick, there are no limits on constraints to add. The current implementation
+Now you got the trick, there are (almost) no limits on constraints to add. The current implementation
 includes for instance _leader preservation_, i.e. the preferred leader has more weight
 than the other partitions.
 
@@ -180,7 +184,19 @@ bin
 t1b1p0, t1b1p0_l, ... , t1b32p9, t1b32p9_l;
 ```
 
-# Usage
+# Real World Usage
+
+Kafka Partitions Assignment Optimizer is public with ❤ by the DAPLAB: [https://kafka-optimizer.daplab.ch/](https://kafka-optimizer.daplab.ch/).
+
+API endpoint: **https://kafka-optimizer.daplab.ch/submit**
+
+## Set `$ZK`
+
+In order to run the below example seamlessly, set the zookeeper server(s):
+
+```
+ZK=daplab-wn-22.fri.lan:2181
+```
 
 ## Retrieve current assignment
 
@@ -189,7 +205,7 @@ $ echo '{"topics": [{"topic": "public.tweets"},{"topic": "trumpet"}], "version":
 ```
 
 ```
-$ /usr/hdp/current/kafka-broker/bin/kafka-reassign-partitions.sh  --zookeeper daplab-wn-22.fri.lan:2181 --generate --topics-to-move-json-file topics-to-move.json --broker-list 0,1,2,3
+$ /usr/hdp/current/kafka-broker/bin/kafka-reassign-partitions.sh  --zookeeper $ZK --generate --topics-to-move-json-file topics-to-move.json --broker-list 0,1,2,3
 Current partition replica assignment
 
 {"version":1,"partitions":[{"topic":"public.tweets","partition":6,"replicas":[1,3]},{"topic":"public.tweets","partition":5,"replicas":[0,2]},{"topic":"public.tweets","partition":0,"replicas":[3,0]},{"topic":"trumpet","partition":0,"replicas":[1,3,0]},{"topic":"public.tweets","partition":3,"replicas":[2,3]},{"topic":"public.tweets","partition":8,"replicas":[3,2]},{"topic":"public.tweets","partition":7,"replicas":[2,0]},{"topic":"public.tweets","partition":1,"replicas":[0,1]},{"topic":"public.tweets","partition":2,"replicas":[1,2]},{"topic":"public.tweets","partition":9,"replicas":[0,3]},{"topic":"public.tweets","partition":4,"replicas":[3,1]}]}
@@ -200,39 +216,68 @@ Proposed partition reassignment configuration
 
 ## Generate REST payload
 
-Copy the `Current partition replica assignment` part and past it in the
+Copy the `Current partition replica assignment` part of the above output and paste it in the
 `partitions` attribute in the `payload.json` file, i.e. something like: 
 
-* `brokers` attribute is of the format: `brokerId[:rack][,brokerId[:rack]]*`, 
-  i.e. a list of comma-separated broker ids and optional `:rack` assignment. 
-* `partitions` attribute is a copy-paste of the `kafka-reassign-partitions` command
-
 ```
 {
-    "brokers": "0:tor2,1:tor1,2:tor2,3:tor1",
+    "brokers": "0:a,1:b,2:a,3:b",
     "partitions": {"version":1,"partitions":[{"topic":"public.tweets","partition":6,"replicas":[1,3]},{"topic":"public.tweets","partition":5,"replicas":[0,2]},{"topic":"public.tweets","partition":0,"replicas":[3,0]},{"topic":"trumpet","partition":0,"replicas":[1,3,0]},{"topic":"public.tweets","partition":3,"replicas":[2,3]},{"topic":"public.tweets","partition":8,"replicas":[3,2]},{"topic":"public.tweets","partition":7,"replicas":[2,0]},{"topic":"public.tweets","partition":1,"replicas":[0,1]},{"topic":"public.tweets","partition":2,"replicas":[1,2]},{"topic":"public.tweets","partition":9,"replicas":[0,3]},{"topic":"public.tweets","partition":4,"replicas":[3,1]}]}
 }
 ```
 
+* `brokers` attribute is of the format: `brokerId[:rack][,brokerId[:rack]]*`, 
+  i.e. a list of comma-separated broker ids and optional `:rack` assignment. 
+* `partitions` attribute is a copy-paste of the `kafka-reassign-partitions` command
+
 # Call the REST API
 
 POST the previously generated payload:
 
 ```
-$ curl -X POST --data @payload.json http://localhost:4567/submit
+$ curl -X POST --data @payload.json https://kafka-optimizer.daplab.ch/submit
 {"version":1,"partitions":[{"topic":"public.tweets","partition":4,"replicas":[3,2]},{"topic":"public.tweets","partition":5,"replicas":[0,1]},{"topic":"public.tweets","partition":6,"replicas":[1,0]},{"topic":"public.tweets","partition":7,"replicas":[2,1]}]}
 ```
 
-You can now copy the output and paste it into `reassignment-file.json` file and call
+You can now copy the output of the command above
+and paste it into `reassignment-file.json` file and call:
+
+```
+/usr/hdp/current/kafka-broker/bin/kafka-reassign-partitions.sh --zookeeper $ZK --reassignment-json-file reassignment-file.json -execute
+```
+
+You can now verify the re-assignment calling:
 
 ```
-kafka-reassign-partitions --zookeeper $ZK --reassignment-json-file reassignment-file.json -execute
+/usr/hdp/current/kafka-broker/bin/kafka-reassign-partitions.sh --zookeeper $ZK --reassignment-json-file reassignment-file.json -verify
 ```
 
+# Increase Number of Replicas
+
+Another common use case is the increase of the replication factor.
+This can be done quickly in setting the attribute `newReplicationFactor` in the payload
+```
+{
+    "brokers": "0,1,2,3,4,5,6,7",
+    "partitions": {"version":1,"partitions":[{"topic":"public.tweets","partition":6,"replicas":[1,3]},{"topic":"public.tweets","partition":5,"replicas":[0,2]},{"topic":"public.tweets","partition":0,"replicas":[3,0]},{"topic":"trumpet","partition":0,"replicas":[1,3,0]},{"topic":"public.tweets","partition":3,"replicas":[2,3]},{"topic":"public.tweets","partition":8,"replicas":[3,2]},{"topic":"public.tweets","partition":7,"replicas":[2,0]},{"topic":"public.tweets","partition":1,"replicas":[0,1]},{"topic":"public.tweets","partition":2,"replicas":[1,2]},{"topic":"public.tweets","partition":9,"replicas":[0,3]},{"topic":"public.tweets","partition":4,"replicas":[3,1]}]},
+    "newReplicationFactor": 3
+}
+```
+
+Call again the service:
+
+```
+$ curl -X POST --data @payload.json https://kafka-optimizer.daplab.ch/submit
+{"version":1,"partitions":[{"topic":"public.tweets","partition":0,"replicas":[3,2,0]},{"topic":"public.tweets","partition":1,"replicas":[0,2,1]},{"topic":"public.tweets","partition":2,"replicas":[1,3,2]},{"topic":"public.tweets","partition":3,"replicas":[2,3,1]},{"topic":"public.tweets","partition":4,"replicas":[3,0,1]},{"topic":"public.tweets","partition":5,"replicas":[0,2,1]},{"topic":"public.tweets","partition":6,"replicas":[1,0,3]},{"topic":"public.tweets","partition":7,"replicas":[2,1,0]},{"topic":"public.tweets","partition":8,"replicas":[3,0,2]},{"topic":"public.tweets","partition":9,"replicas":[0,1,3]}]}
+```
+
+All the partitions have 3 replicas now, and the existing ones are preserved as much as possible (in this example, all the existing replicas are preserved, some leader have been changed though).
+
+
 # No changes
 
-If the current assignment is already optimal, the API will simply answer with an empty list,
-as follow:
+Please note that the API do return only the changes. If the current assignment is already optimal, 
+the API will simply answer with an empty list, as follow:
 
 ```
 {"version":1,"partitions":[]}