Skip to content

Commit e68e293

Browse files
committed
Add more documentation details
1 parent 5f7bf1c commit e68e293

File tree

1 file changed

+25
-18
lines changed

1 file changed

+25
-18
lines changed

README.md

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
Kafka Partitions Assignment Optimizer
22
====
33

4-
If you have more than 4 brokers spread on several top-of-rack switches (TOR),
4+
If you have more than 4 brokers spread on several top-of-rack switches (_TOR_),
55
you might be interested in balancing replicas and leaders properly to
66
survive to a switch failure and to avoid bottlenecks.
77

@@ -18,13 +18,13 @@ overall cluster availability.
1818

1919
Also, if you running a version of Kafka which does not include
2020
[KIP-36 (rack aware replica assignment)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment))
21-
you don't have any knowledge of the network topology in the
21+
you don't have any knowledge about the network topology in the
2222
assignment algorithm.
2323

2424
## Demonstration: `kafka-reassign-partitions.sh` under-efficiency
2525

26-
Assume have a cluster with 20 brokers, named 0-19, spread across 2 switches.
27-
Brokers with odd numbers are all on the same TOR `tor1`,
26+
Lets assume we have a cluster with 20 brokers, named 0-19, spread across 2 switches.
27+
Brokers with odd numbers are all on the same _TOR_ `tor1`,
2828
brokers with even numbers are wired to `tor2`.
2929

3030
We have a topic `x.y.z.t` with 10 partitions and a replication factor of 2.
@@ -77,10 +77,11 @@ Proposed partition reassignment configuration
7777
]}
7878
```
7979

80-
(I did just re-format and sort the json output for sake of clarity).
80+
_(I did just re-format and sort the json output for sake of clarity)._
8181

8282
If you compare partition by partition, you can see a **lot** of changes in the partition assignment.
83-
When computing the diff manually, we could simply change the assignment of partition `1`, like
83+
That's rather unfortunate, since computing the diff manually,
84+
we could simply change the assignment of partition `1`, like:
8485

8586
```
8687
{"topic":"x.y.z.t","partition":1,"replicas":[8,1]},
@@ -91,30 +92,32 @@ All the other moves are not required.
9192
Of course `kafka-reassign-partitions` is only proposing an example reassignment
9293
configuration and editing manually might appear easy,
9394
but when you're dealing with bigger topics with 40 or more partitions
94-
and you're under fire, you'd like to have a tool which is doing that for you properly
95-
without too many manual edits.
95+
and you're under fire, you'd probably like to have a tool
96+
on which you can rely to do that right without too many manual edits.
9697

9798
LinkedIn open-sourced its [kafka-tools](https://github.com/linkedin/kafka-tools)
9899
which has really nice features for day to day operations, but lots of
99100
`random.shuffle(replicas)` are used internally, which might end-up in
100-
sub-optimal placements. The tool don't have rack awareness either.
101+
sub-optimal placements. The tool don't have rack awareness either at the time
102+
of writing.
101103

102104

103-
# Replica assignment as an optimization function
104-
105+
# Replica assignment as a constraint satisfaction problem
106+
105107
If you think out of the box, replicas assignments looks like an
106108
[optimization function](https://en.wikipedia.org/wiki/Mathematical_optimization)
107-
under specific constraints.
108-
109+
under specific constraints, or a
110+
[constraint satisfaction problem](https://en.wikipedia.org/wiki/Constraint_satisfaction_problem)
109111
For instance, "no two replicas of the same partition assigned to the same broker" is one of
110-
these constraint.
112+
these constraints.
111113

112114
To minimize the move of replicas, the idea is to assign more weight (i.e. more value)
113115
to existing assignments, so that the linear optimization will try to preserve
114116
existing assignment (and in turn minimising the number of bytes moved across the brokers).
115117

116118
Let's define a variable as a concatenation of broker id and partition id, such as
117-
`b9_p6`. This variable will be 1 if the partition 6 is assigned to the broker 9.
119+
`b9_p6`. This variable will be 1 if the partition 6 is assigned to the broker 9,
120+
0 otherwise.
118121

119122
The previous constraint, "no two replicas of the same partition assigned to the same broker",
120123
would be expressed as
@@ -125,9 +128,8 @@ Now you got the trick, there are no limits on constraints to add. The current im
125128
includes for instance _leader preservation_, i.e. the preferred leader has more weight
126129
than the other partitions.
127130

128-
[lp_solve]() is used in the background to solve the linear equation generated.
129-
130-
131+
[lp_solve](http://lpsolve.sourceforge.net/5.5/) is used behind the scene
132+
to solve the generated linear equation.
131133

132134

133135
## Example of equation
@@ -234,3 +236,8 @@ If no change, the API call answers:
234236
```
235237
{"version":1,"partitions":[]}
236238
```
239+
240+
# Greetings
241+
242+
* http://www.hostmath.com/ for the equation graphics
243+

0 commit comments

Comments
 (0)