Ellis MichaelPh.D. Student in Computer Science at the University of Washington
https://ellismichael.com/
Fri, 21 Jun 2019 00:43:27 -0700Fri, 21 Jun 2019 00:43:27 -0700Jekyll v3.8.3Randomized Consensus on Unknown Domains in Finite Rounds<p>Warning: This post contains LaTeX rendered with MathJax. It may not render properly in your RSS reader.</p><p>The most widely known result in distributed computing is the FLP result that
proves the impossibility of consensus <a href="#flp">[1]</a>. One of the assumptions of
this theorem is that processes are <em>deterministic</em>. When randomness is allowed,
the Ben-Or algorithm gives simple, elegant way to solve binary consensus <a href="#ben-or83:anot-adva">[2]</a>. Or rather, <script type="math/tex">\text{lim}_{t \to \infty} P(\text{consensus
has been reached}) = 1</script>. The algorithm goes something like this.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a = input({0, 1})
loop:
send_phase1(a)
A = receive_phase1()
if ∃aʹ∈ A : |{a''∈ A : a'' = a'}| > n/2:
b = a'
else:
b = ⊥
send_phase2(b)
B = receive_phase2()
if ∃bʹ∈ B : b' != ⊥ && |{b'' ∈ B : b'' = b'}| > f:
decide(bʹ)
if ∃bʹ∈ B : b' != ⊥:
a = b'
else:
a = choose_random({0, 1})
</code></pre></div></div>
<p>The algorithm proceeds in asynchronous rounds, each with two phases. In each
phase, each process broadcasts its value for the phase and then waits for the
values of <script type="math/tex">n - f</script> other processes. The safety argument is not complicated,
and the termination argument is even more straightforward. Eventually the random
choices made by processes in some round will be overwhelmingly in favor of
either 0 or 1, and that value will decided by every process in the next round.</p>
<p>This blog post is a record of a couple simple observations I made when
re-reading the paper recently (in preparation for teaching it in
<a href="https://courses.cs.washington.edu/courses/cse452/19sp/">UW’s distributed systems course</a>
this past quarter) that I couldn’t find written up anywhere else.</p>
<h2>Unknown Domains</h2>
<p>The first observation is that this algorithm can be modified to allow processes
to have inputs values from larger domains than <script type="math/tex">\{0, 1\}</script>. In fact, the input
domain can be infinite. Furthermore, we don’t even need the processes to know
the input domain <em>a priori</em>. On the very last line of the protocol, when
processes make a random choice, they simply choose from <em>all values seen so
far</em> in any message.</p>
<p>Since there are at most <em>n</em> possible input values (because there are at most <em>n</em>
processes), the termination argument still applies. The only real difference is
that termination could take longer. The original paper contained a theorem that
when <script type="math/tex">f</script> is <script type="math/tex">O(\sqrt{n})</script>, the expected number of rounds is constant. That
theorem no longer applies when the set of possible decision values grows with <script type="math/tex">n</script>.</p>
<p>The other interesting fact is that this protocol can be further modified to
support a model where not all processes get input values. Processes without
input values initialize <script type="math/tex">a = \bot</script> on the first line. And on the last line
when random choices are made, processes choose from all non-<script type="math/tex">\bot</script> values
seen. As long as at least <script type="math/tex">f + 1</script> processes get input values, termination is
still guaranteed. Even more interestingly, the check on first-phase messages can
be refined somewhat in this case. The property that must be maintained is that
no second-phase messages with different non-<script type="math/tex">\bot</script> values should exist for the
same round. So, assuming we get <script type="math/tex">n - f</script> messages in the second round, if the
number of messages for one non-<script type="math/tex">\bot</script> value is more than <script type="math/tex">f</script> greater
than for every other non-<script type="math/tex">\bot</script> value, then we can safely choose that for the
second phase.</p>
<p>Both of these modifications are important for the way people use consensus in
the real world — namely to implement state machine replication. The commands
proposed to the state machine log often come from an infinite domain and are
chosen by clients at runtime. The servers implementing the state machine do not
know which commands will be proposed <em>a priori</em>. Furthermore, we would like the
system to make progress even when clients don’t send their commands to all
servers.</p>
<h2>Finite Rounds</h2>
<p>The other observation is that the original protocol had all processes taking
protocol steps and sending messages forever, even after they had decided values.
This is not necessary. Instead, we can have processes send messages for each
phase <em>proactively</em> up until they’ve decided a value and the <em>reactively</em>
thereafter. That is, once a process has reached a decision, it only sends a
message for a phase after it has received at least one message for that phase
(or a later phase).</p>
<p>Because all non-faulty processes eventually decide a value, eventually all
processes stop sending messages. Note, however, that this does not mean
processes can halt. They must continue listening for messages forever (or at
least <script type="math/tex">f+1</script> processes must). Eventually, however, they will no longer send any
new messages.</p>
<p>When combined the model above where not all processes get input values, we could
also specify that processes which have not seen any non-<script type="math/tex">\bot</script> values remain
in a reactive mode until they receive a non-<script type="math/tex">\bot</script> value.</p>
<h3>References</h3>
<ol class="acm-bib"><li><span id="flp"><span style="font-variant: small-caps">Fischer, M.J., Lynch, N.A. and Paterson, M.S.</span> Impossibility of Distributed Consensus with One Faulty Process. <i>J. ACM</i>. 32(2):374–382. Apr. 1985.</span></li>
<li><span id="ben-or83:anot-adva"><span style="font-variant: small-caps">Ben-Or, M.</span> Another Advantage of Free Choice (Extended Abstract): Completely Asynchronous Agreement Protocols. In <i>Proceedings of the Second Annual ACM Symposium on Principles of Distributed Computing</i>. Montreal, Quebec, Canada. 1983. 27–30.</span></li></ol>
Mon, 10 Jun 2019 00:00:00 -0700
https://ellismichael.com/blog/2019/06/10/randomized-consensus/
https://ellismichael.com/blog/2019/06/10/randomized-consensus/technicalUnderstanding Traces<p>Warning: This post contains LaTeX rendered with MathJax. It may not render properly in your RSS reader.</p><p>For <a href="https://github.com/emichael/dslabs">DSLabs</a>, one of the pieces I built was a
<a href="https://github.com/emichael/dslabs/blob/409be9bb71dc0844f24f502d658a47bf1bd9715b/framework/tst/dslabs/framework/testing/search/SearchState.java#L329">post-processing phase</a> that runs on traces generated by the
model checker. It takes a trace (i.e., a list of events) and produces a
semantically equivalent trace which is, hopefully, more understandable to
humans.</p>
<p>To motivate the problem, lets look at an example. Suppose we have a distributed
system with four processes — <script type="math/tex">p_1</script>, <script type="math/tex">p_2</script>, <script type="math/tex">p_3</script>, and <script type="math/tex">p_4</script>,
and consider the events represented by the following space-time diagram:</p>
<p><img src="/img/tikz/d4df8b5866db57366e979b530d98ea40.svg" class="tikz" style="width: 300px;" /></p>
<p>Because there are <em>concurrent</em> events (i.e., events not ordered by the
happens-before relation) in the above execution, there is more than way to
represent these events as a well-formed trace.<sup id="fnref:num-traces"><a href="#fn:num-traces" class="footnote">1</a></sup> Here, a well-formed
trace is one in which the happens-before relation is respected by the ordering
of events. However, not all traces are created equal. Without knowing the
specifics of the protocol being run in this execution, I think most people, if
asked to produce a trace, would write something like this:</p>
<ul>
<li><script type="math/tex">p_1</script> sends <script type="math/tex">m_1</script> to <script type="math/tex">p_2</script></li>
<li><script type="math/tex">p_2</script> receives <script type="math/tex">m_1</script>, sends <script type="math/tex">m_2</script> to <script type="math/tex">p_1</script></li>
<li><script type="math/tex">p_1</script> receives <script type="math/tex">m_2</script>, sends <script type="math/tex">m_3</script> to <script type="math/tex">p_2</script></li>
<li><script type="math/tex">p_2</script> receives <script type="math/tex">m_3</script></li>
<li><script type="math/tex">p_3</script> sends <script type="math/tex">m_4</script> to <script type="math/tex">p_4</script></li>
<li><script type="math/tex">p_4</script> receives <script type="math/tex">m_4</script>, sends <script type="math/tex">m_5</script> to <script type="math/tex">p_3</script></li>
<li><script type="math/tex">p_3</script> receives <script type="math/tex">m_5</script>, sends <script type="math/tex">m_6</script> to <script type="math/tex">p_4</script></li>
<li><script type="math/tex">p_4</script> receives <script type="math/tex">m_6</script></li>
</ul>
<p>They might, instead, pair up <script type="math/tex">m_1</script> and <script type="math/tex">m_4</script>, <script type="math/tex">m_2</script> and <script type="math/tex">m_5</script>,
and <script type="math/tex">m_3</script> and <script type="math/tex">m_6</script>. But they certainly wouldn’t write down the
following:</p>
<ul>
<li><script type="math/tex">p_1</script> sends <script type="math/tex">m_1</script> to <script type="math/tex">p_2</script></li>
<li><script type="math/tex">p_3</script> sends <script type="math/tex">m_4</script> to <script type="math/tex">p_4</script></li>
<li><script type="math/tex">p_2</script> receives <script type="math/tex">m_1</script>, sends <script type="math/tex">m_2</script> to <script type="math/tex">p_1</script></li>
<li><script type="math/tex">p_1</script> receives <script type="math/tex">m_2</script>, sends <script type="math/tex">m_3</script> to <script type="math/tex">p_2</script></li>
<li><script type="math/tex">p_4</script> receives <script type="math/tex">m_4</script>, sends <script type="math/tex">m_5</script> to <script type="math/tex">p_3</script></li>
<li><script type="math/tex">p_2</script> receives <script type="math/tex">m_3</script></li>
<li><script type="math/tex">p_3</script> receives <script type="math/tex">m_5</script>, sends <script type="math/tex">m_6</script> to <script type="math/tex">p_4</script></li>
<li><script type="math/tex">p_4</script> receives <script type="math/tex">m_6</script></li>
</ul>
<p>Not just that, but the latter trace is also harder to follow as a reader. It has
several unnecessary “context switches.” The DSLabs model checker, however, does
not distinguish between these two traces; they both have the same length. If an
invariant was violated upon completion of the execution in question, the
concurrent events in the trace produced by the model checker would be randomly
ordered.</p>
<p>The post-processing phase I implemented is based on a simple idea, that events
in traces should be followed by their immediate consequences. First, I took the
trace produced by the model checker and built up the associated event graph. The
event graph looks just like the space-time diagram above. Vertices are events;
edges go between corresponding <em>send</em> and <em>receive</em> events and between
subsequent events at the same process.</p>
<p>Then, I ran a depth-first topological sort of the event graph. By topologically
sorting the graph, we get a well-formed trace. By doing it in a depth-first
manner, we list events and their immediate consequences whenever possible.</p>
<p>To see what this looks like in practice, let’s look at an example from the
primary-backup lab. First, we show an animated trace demonstrating an
invariant-violation in a buggy solution.</p>
<video controls="" loop="loop" preload="metadata">
<source src="/img/posts/understanding-traces/ugly-trace.webm#t=0.1" type="video/webm" />
<source src="/img/posts/understanding-traces/ugly-trace.mp4#t=0.1" type="video/mp4" />
</video>
<p>Next, let’s see what the same invariant-violation looks with the post-processing
phase for understandability.</p>
<video controls="" loop="loop" preload="metadata">
<source src="/img/posts/understanding-traces/good-trace.webm#t=0.1" type="video/webm" />
<source src="/img/posts/understanding-traces/good-trace.mp4#t=0.1" type="video/mp4" />
</video>
<p>Of course, you can’t necessarily follow what’s going on without knowing the
details of the protocol. But, it should be much easier to follow the “flow” of
the second video.</p>
<p>There are many possibilities for further improvement. You could bias the sorting
of events towards message delivery or towards staying with the same process. You
could even try to recognize group communication patterns (e.g., the
“one→all→one” pattern) and order them in particular ways. For now, though, I think this solution works quite well.</p>
<div class="footnotes">
<ol>
<li id="fn:num-traces">
<p>In fact, there are 20 such traces. <a href="#fnref:num-traces" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Mon, 18 Feb 2019 00:00:00 -0800
https://ellismichael.com/blog/2019/02/18/understanding-traces/
https://ellismichael.com/blog/2019/02/18/understanding-traces/technicalProducers and Consumers<p>Warning: This post contains LaTeX rendered with MathJax. It may not render properly in your RSS reader.</p><p>At a recent talk of Leslie Lamport, he raised the question of how many possible executions there are for a bounded-buffer producer-consumer protocol for a given set of input values.
Leslie’s point was that, from the standpoint of the values produced and consumed, there is only one execution, and the only things that we can say about the ordering of these events is that a value must be produced before it is consumed and that before a value can be produced, enough values must have been consumed so that there is room in the buffer.</p>
<p>I started wondering, though, just how many executions are there exactly?
For a given buffer size and length of execution, how many possible (legal) sequences of produce and consume events are possible?
Equivalently, how many strings of <script type="math/tex">P</script>s and <script type="math/tex">C</script>s of a given length are there such that the number of <script type="math/tex">C</script>s in any prefix is less than or equal to the number of <script type="math/tex">P</script>s and the number of <script type="math/tex">P</script>s minus the number of <script type="math/tex">C</script>s in any prefix is less than or equal to the buffer size?
Writing down the recurrence for the general case is easy enough, and computers are very fast and good with numbers, so let’s see a few results.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">functools</span>
<span class="nd">@functools.lru_cache</span><span class="p">(</span><span class="n">maxsize</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">B</span><span class="p">,</span> <span class="n">S</span><span class="p">,</span> <span class="n">U</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span>
<span class="k">if</span> <span class="n">U</span> <span class="o"><</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">U</span> <span class="o">></span> <span class="n">B</span><span class="p">:</span> <span class="k">return</span> <span class="mi">0</span>
<span class="k">if</span> <span class="n">S</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> <span class="k">return</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">f</span><span class="p">(</span><span class="n">B</span><span class="p">,</span> <span class="n">S</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">U</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="n">f</span><span class="p">(</span><span class="n">B</span><span class="p">,</span> <span class="n">S</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">U</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
</code></pre></div></div>
<p>Here, <script type="math/tex">B</script> is the size of the buffer, <script type="math/tex">S</script> is the number of steps in the execution, and <script type="math/tex">U</script> is the number of used slots in the buffer (initially 0).</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">print</span><span class="p">([</span> <span class="n">f</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="p">])</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
</code></pre></div></div>
<p>When the buffer size is 1, there only ever one execution, no matter how many steps we take.
It’s just the <script type="math/tex">S</script>-length prefix of the infinite <script type="math/tex">PCPCPCPCPC...</script> string.
What about when the buffer size is 2?</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">print</span><span class="p">([</span> <span class="n">f</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="p">])</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">128</span><span class="p">]</span>
</code></pre></div></div>
<p>Here, we see once-repeated increasing powers of 2.
To see why this is the case, imagine the state of the buffer.
When it’s empty or full, there is only one action that can be taken, a produce or consume, respectively.
On the other hand, when there is a single value in the buffer, both actions are possible.</p>
<p><img src="/img/tikz/4147538e228d27ec2d6fa9aa7cb0555d.svg" class="tikz" /></p>
<p>Now, given both of those results, try to guess what the sequence looks like when <script type="math/tex">B = 3</script>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">print</span><span class="p">([</span> <span class="n">f</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="p">])</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">13</span><span class="p">,</span> <span class="mi">21</span><span class="p">,</span> <span class="mi">34</span><span class="p">,</span> <span class="mi">55</span><span class="p">,</span> <span class="mi">89</span><span class="p">,</span> <span class="mi">144</span><span class="p">,</span> <span class="mi">233</span><span class="p">,</span> <span class="mi">377</span><span class="p">,</span> <span class="mi">610</span><span class="p">]</span>
</code></pre></div></div>
<p>Did you guess the Fibonacci sequence?
This one’s a little harder to see.
It’s not obvious (to me at least) why the Fibonacci recurrence should apply.
If <script type="math/tex">f(3, i)</script> is the number of executions possible with exactly <script type="math/tex">i</script> steps with a buffer of size 3, let’s let <script type="math/tex">f_{0, 3}(3, i)</script> be the number of those executions where the final state of the buffer is full or empty.
Then, let <script type="math/tex">f_{1, 2}(3, i)</script> be the number of executions ending in the buffer having 1 or 2 values.
First, we have the obvious identity.</p>
<script type="math/tex; mode=display">f(3, i) = f_{0, 4}(3, i) + f_{1, 2}(3, i)</script>
<p>Then, we have a pair of recurrences that are easiest to see pictorially.</p>
<p><img src="/img/tikz/7a4dadb06a942bc92d9345757c23a284.svg" class="tikz" /></p>
<p>Namely, the following:</p>
<script type="math/tex; mode=display">f_{0, 4}(3, i) = f_{1, 2}(3, i - 1)</script>
<script type="math/tex; mode=display">f_{1, 2}(3, i) = f_{0, 4}(3, i - 1) + f_{1, 2}(3, i - 1)</script>
<p>Then, the derivation is straightforward.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
f(3, i) &= f_{0, 4}(3, i) + f_{1, 2}(3, i) \\
&= f_{0, 4}(3, i - 1) + 2 f_{1, 2}(3, i - 1) \\
&= f(3, i - 1) + f_{1, 2}(3, i - 1) \\
&= f(3, i - 1) + f_{0, 4}(3, i - 2) + f_{1, 2}(3, i - 2) \\
&= f(3, i - 1) + f(3, i - 2)
\end{align*} %]]></script>
<p>Finally, let’s see what happens when the buffer is unbounded (by setting the size of the buffer to the number of steps in the execution).</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">print</span><span class="p">([</span> <span class="n">f</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="p">])</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">35</span><span class="p">,</span> <span class="mi">70</span><span class="p">,</span> <span class="mi">126</span><span class="p">,</span> <span class="mi">252</span><span class="p">,</span> <span class="mi">462</span><span class="p">,</span> <span class="mi">924</span><span class="p">,</span> <span class="mi">1716</span><span class="p">,</span> <span class="mi">3432</span><span class="p">]</span>
</code></pre></div></div>
<p>Here, we see a different sequence.
The number of possible executions with <script type="math/tex">i</script> steps is <script type="math/tex">{i \choose \lfloor i / 2 \rfloor}</script>.
Let <script type="math/tex">X</script> be set of <script type="math/tex">i</script>-length <script type="math/tex">P/C</script> sequences with exactly <script type="math/tex">\left \lfloor \frac{i}{2} \right \rfloor</script> <script type="math/tex">C</script>s (some of which represent invalid executions) and let <script type="math/tex">Y</script> be the set of valid sequences.</p>
<p>First, given a sequence in <script type="math/tex">X</script>, we can transform it into a valid sequence in <script type="math/tex">Y</script> by scanning left to right, keeping track the number of values in the buffer.
If a <script type="math/tex">C</script> is invalid (i.e., would result in the number of values in the buffer becoming negative), we change it to a <script type="math/tex">P</script> and <em>treat it as a null operation on the buffer</em> (i.e., neither a <script type="math/tex">C</script> nor a <script type="math/tex">P</script>).
So, <script type="math/tex">CPCCPPPP</script> would become <script type="math/tex">PPCPPPPP</script>.
Then, to invert that transformation, we count the number of <script type="math/tex">P</script>s that should be transformed to <script type="math/tex">C</script>s and then repeatedly scan from right to left, replacing a <script type="math/tex">P</script> with a <script type="math/tex">C</script> every time doing so would cause the sequence to become invalid, treating the new <script type="math/tex">C</script> as neither a <script type="math/tex">P</script> nor <script type="math/tex">C</script> for the remainder of the operation.
Here are these two transformations in Python.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">t</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="n">pc</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">r</span> <span class="o">=</span> <span class="s">''</span>
<span class="k">for</span> <span class="n">xi</span> <span class="ow">in</span> <span class="n">x</span><span class="p">:</span>
<span class="k">if</span> <span class="n">pc</span> <span class="o"><=</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">xi</span> <span class="o">==</span> <span class="s">'C'</span><span class="p">:</span>
<span class="n">r</span> <span class="o">+=</span> <span class="s">'P'</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">r</span> <span class="o">+=</span> <span class="n">xi</span>
<span class="k">if</span> <span class="n">xi</span> <span class="o">==</span> <span class="s">'P'</span><span class="p">:</span>
<span class="n">pc</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">pc</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">r</span>
<span class="k">def</span> <span class="nf">ti</span><span class="p">(</span><span class="n">y</span><span class="p">):</span>
<span class="n">y</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">y</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="s">'P'</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="o">-</span> <span class="n">p</span>
<span class="k">while</span> <span class="n">p</span> <span class="o">></span> <span class="nb">len</span><span class="p">(</span><span class="n">y</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="p">:</span>
<span class="n">pi</span><span class="p">,</span> <span class="n">ci</span> <span class="o">=</span> <span class="n">p</span><span class="p">,</span> <span class="n">c</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">):</span>
<span class="k">if</span> <span class="n">y</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">==</span> <span class="s">'C'</span><span class="p">:</span>
<span class="n">ci</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="n">y</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">==</span> <span class="s">'P'</span><span class="p">:</span>
<span class="n">pi</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="n">ci</span> <span class="o">>=</span> <span class="n">pi</span><span class="p">:</span>
<span class="n">p</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="n">y</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="s">'X'</span>
<span class="k">break</span>
<span class="k">return</span> <span class="s">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s">'C'</span> <span class="k">if</span> <span class="n">yi</span> <span class="o">==</span> <span class="s">'X'</span> <span class="k">else</span> <span class="n">yi</span> <span class="k">for</span> <span class="n">yi</span> <span class="ow">in</span> <span class="n">y</span><span class="p">)</span>
</code></pre></div></div>
<p>Since these transformations are, indeed, inverses of each other, we get
<script type="math/tex">|X| = |Y|</script>.</p>
<p>Of course, none of the above observations are new, and some/all are simple enough to be put to undergraduates in a discrete math course.
Typically, these problems would be put in terms of the number of sequences of 1 and -1 where the partial sums are all nonnegative and less than some bound.
I just liked that you could describe interesting combinatorics problems in terms of a common synchronization primitive.</p>
<p>Furthermore, there are still more interesting sequences for different values of <script type="math/tex">B</script>.
For instance, when <script type="math/tex">B = 4</script>, we get:</p>
<script type="math/tex; mode=display">% <![CDATA[
f(4, i) = \begin{cases}
1 & i = 0 \\
3^{\frac{i - 1}{2}} & \text{$i$ odd} \\
2 \cdot 3^{\frac{i}{2} - 1} & \text{$i$ even}, i \ne 0
\end{cases} %]]></script>
<p>If anyone knows a closed-form expression for <script type="math/tex">f (B, S)</script>, I would be grateful.</p>
<h3>Acknowledgments</h3>
<p>Thanks to <a href="https://homes.cs.washington.edu/~helgi/">Helgi</a> for pointing out a bug in an earlier version of my <code class="highlighter-rouge">t(x)</code> and <code class="highlighter-rouge">ti(y)</code> functions.</p>
Mon, 27 Aug 2018 00:00:00 -0700
https://ellismichael.com/blog/2018/08/27/num-executions-producers-consumers/
https://ellismichael.com/blog/2018/08/27/num-executions-producers-consumers/technicalA Game of Thrones<p>Warning: This post contains LaTeX rendered with MathJax. It may not render properly in your RSS reader.</p><p>Suppose you’re the ruler of a vast empire, composed of <script type="math/tex">k</script> kingdoms.
As emperor, you’ve decided that who sits on each throne is too important to be left up to random things like birth order; you’d like to manage the succession to each throne personally.
You don’t want to be bothered each time a king or queen dies, however.
You are emperor, after all.</p>
<p>Instead, you decide that you will handpick the line of succession for each of the <script type="math/tex">k</script> thrones.
These succession orders will be valid until <script type="math/tex">f</script> nobles sitting on or in line to a throne die, at which point you will revisit the succession question and design a new chart.
There is one caveat, however: you do not want any of the kingdoms to be consolidated.
None of your vassals should sit on two thrones simultaneously.</p>
<p>Let <script type="math/tex">n</script> be the number of your vassals you consider worthy of sitting on one of your empire’s thrones.
We’d like to answer the following question:
how large must <script type="math/tex">n</script> be to support lines of succession for <script type="math/tex">k</script> thrones such that no king or queen will ever sit on two thrones even if <script type="math/tex">f</script> nobles in those succession lines die?</p>
<p>In order to have a suitable succession chart, what you need to come up with is a matrix:</p>
<script type="math/tex; mode=display">% <![CDATA[
%--------------------
% PREAMBLE
%--------------------
\require{cancel}
\require{color}
\renewcommand\CancelColor{\color{blue}}
\newcommand{\ullabeld}[3]{ {\scriptstyle #3} \left \{
\vphantom{#1}\smash{\overbrace{#1}^{#2}}
\right.
\vphantom{\overbrace{#1}^#2}
}
%--------------------
% END PREAMBLE
%--------------------
\ullabeld{\begin{matrix}
\nu_{1,1} & \nu_{1,2} & \cdots & \nu_{1,k} \\
\nu_{2,1} & \nu_{2,2} & \cdots & \nu_{2,k} \\
\vdots \\
\nu_{f+1,1} & \nu_{f+1,2} & \cdots & \nu_{f+1,k}
\end{matrix}}{k}{f+1} %]]></script>
<p>where <script type="math/tex">\nu_{i,j}</script> is the <script type="math/tex">i</script>th in line to the <script type="math/tex">k</script>th throne (the 1st in line currently sits on the throne).
For a given <script type="math/tex">j</script>, each of the <script type="math/tex">\nu_{i, j}</script> must be unique.
This matrix also needs to have the special property that if up to <script type="math/tex">f</script> of the nobles you appoint die, none of your vassals occupies two thrones simultaneously.</p>
<p>For instance, a succession setup for <script type="math/tex">k=3</script>, <script type="math/tex">f=3</script> of the following form is not admissible under the previous condition.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
a & b & c \\
d & e & f \\
g & d & h \\
i & j & k \\
\end{matrix} %]]></script>
<p>If the following 3 nobles died, then <script type="math/tex">d</script> would sit on two thrones at the same time.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
\xcancel{a} & \xcancel{b} & c \\
d & \xcancel{e} & f \\
g & d & h \\
i & j & k \\
\end{matrix} %]]></script>
<p>With the groundwork laid, we’re ready to start tackling the problem of how large <script type="math/tex">n</script> must be to support an admissible succession plan for any given <script type="math/tex">k</script> and <script type="math/tex">f</script>.
The first thing to notice is that none of your vassals currently sitting on a throne can be in the line of succession for any other throne; otherwise, if the <script type="math/tex">f</script> nobles ahead of them in the other line of succession died, they’d sit on two thrones simultaneously.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
v_1 & \xcancel{\cdot} \\
\cdot & \xcancel{\cdot} \\
\cdot & \xcancel{\cdot} \\
\vdots \\
\cdot & v_1 \\
\end{matrix} %]]></script>
<p>So, the picture we have so far looks like this:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
v_1 & v_2 & v_3 & \cdots & v_k \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\vdots \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\end{matrix} %]]></script>
<p>The next thing to notice is that any second-in-line noble can occupy any of the last-in-line positions to any other throne but no higher position.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
v_1 & v_2 & v_3 & \cdots & v_k \\
v_{k+1} & \cdot & \cdot & \cdots & \cdot \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\vdots \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\cdot & v_{k+1} & \cdot & \cdots & \cdot \\
\end{matrix} %]]></script>
<p>In fact, a second-in-line noble can occupy <em>all</em> of the other last-in-line positions.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
v_1 & v_2 & v_3 & \cdots & v_k \\
v_{k+1} & \cdot & \cdot & \cdots & \cdot \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\vdots \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\cdot & v_{k+1} & v_{k+1} & \cdots & v_{k+1} \\
\end{matrix} %]]></script>
<p>But, we need at least <script type="math/tex">k</script> second-in-line nobles, so we might as well distribute the other positions equally.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
v_1 & v_2 & v_3 & \cdots & v_k \\
v_{k+1} & v_{k+2} & v_{k+3} & \cdots & v_{2k} \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
\vdots \\
\cdot & \cdot & \cdot & \cdots & \cdot \\
v_{2k} & v_{k+1} & v_{k+2} & \cdots & v_{2k -1} \\
\end{matrix} %]]></script>
<p>Similarly, a third-in-line noble can be second-to-last in another line of succession but no higher.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{matrix}
v_1 & v_2 & v_3 & \cdots & v_k \\
v_{k+1} & v_{k+2} & v_{k+3} & \cdots & v_{2k} \\
v_{2k+1} & v_{2k+2} & v_{2k+3} & \cdots & v_{3k} \\
\vdots \\
v_{3k} & v_{2k+1} & v_{2k+2} & \cdots & v_{3k -1} \\
v_{2k} & v_{k+1} & v_{k+2} & \cdots & v_{2k -1} \\
\end{matrix} %]]></script>
<p>This trend continues until we get to the “middle” row (the <script type="math/tex">\left \lceil \frac{f}{2} \right \rceil + 1</script> row). If <script type="math/tex">f</script> is even, then this is the exact same situation as the above.
Nobles in these positions in the lines of successions can also occupy the position directly below this one in another line of succession but cannot simultaneously occupy any positions on the same row or higher.
However, if <script type="math/tex">f</script> is odd, a single noble can simultaneously occupy <em>all</em> positions in this row, since <script type="math/tex">2 \left \lceil \frac{f}{2} \right \rceil > f</script>.</p>
<p>This brings us finally to a lower bound on a value of <script type="math/tex">n</script>, as well as a plan for what the lines of succession could look like when the bound is tight.</p>
<script type="math/tex; mode=display">% <![CDATA[
n \ge \begin{cases}
k\left (\frac{f}{2} + 1 \right ) & \text{$f$ even} \\
k \left (\left \lfloor \frac{f}{2} \right \rfloor + 1 \right ) + 1 & \text{$f$ odd}
\end{cases} %]]></script>
Fri, 12 Jan 2018 00:00:00 -0800
https://ellismichael.com/blog/2018/01/12/a-game-of-thrones/
https://ellismichael.com/blog/2018/01/12/a-game-of-thrones/technicalOn Ink Shortages<p>In the original presentation of the Paxos algorithm, Lamport used an extended metaphor involving the legislators of a bygone civilization <a href="#lamport98:part-time">[1]</a>.
These legislators would communicate by messenger and pass (non-contradictory) laws.
One of the key assumptions of this protocol was that the legislators had special ledgers they carried around and could write on with <em>indelible ink</em>.</p>
<p>This is my (almost certainly misguided) attempt at an allegorical explanation of our recent work on diskless recovery in distributed systems <a href="#recovering-disc17">[2]</a>.</p>
<h2>Naxos</h2>
<p>Having recently undertaken a long archaeological study of my own, I discovered that, shortly after its development, the parliamentary protocol used on Paxos spread throughout the region and was adopted by other parliaments (curiously, always with a few tweaks and always with a letter or two prepended to the name “Paxos”).</p>
<p>The protocol eventually spread to the the far-away (though similarly-named) island of Naxos.
The Naxons, however, were not as wealthy as the Paxons and could not afford the vast quantities of expensive indelible ink the Paxon legislators used to record their decrees.
(All other available inks had a tendency to fade over time and were thus unsuitable for parliamentary business.) However, Naxon legislators—out of necessity—had prodigious memories, and when they committed to a promise during the legislative process, they did not forget it.</p>
<p>At first, the Naxon legislators used the multi-decree Paxos protocol without issue.
This protocol worked perfectly well for a number of years, until the first time a Naxon legislator was replaced.
The Naxon parliament had no term limits, and incumbents almost always won (thanks to massive campaign contributions from the olive oil industry), but when a challenger finally won election for the first time, something happened that had never happened before—the parliament passed contradictory laws.</p>
<h2>The Incident</h2>
<p>As per parliamentary procedure, Naxon legislators used each other’s official titles to conduct all business (e.g., the gentleman from Σεαττλε).
While the aforementioned election was ongoing, the MP from Μουνταιν Ϝιεω attempted to pass a law making Λεσλιε the cheese inspector for the coming year.
Μουνταιν Ϝιεω’s representative was able to secure votes for this decree from a majority of legislators—including MP who was about to be replaced, the MP from Βοστον.</p>
<p>However, this process of gathering votes took quite a bit of time and lasted well past the election.
After he was elected, the new legislator from Βοστον proposed a decree making Βαρβαρα the cheese inspector for the coming year, which was also able to pass.
As you can imagine, this led to some controversy.</p>
<p>The Naxon political scientists were not complete neophytes and knew about the potential for problems when new legislators were elected; they had designed a procedure specifically to handle this process.
Because this procedure had to work in the case of the death of an MP, the Naxon protocol could not rely on the ability of a newly elected legislator to learn all previous decisions from the former legislator of their district.
The political scientists, therefore, decided that newly elected legislators would, upon election, contact a majority of the parliament to learn about previously agreed-to decrees and adopted ballots.</p>
<p>The problem arose during the passing of the decree to make Λεσλιε cheese inspector.
The decree’s sponsor rightly reckoned that the old MP from Βοστον might be soon replaced, and so sent the vote request to him via express courier.
While waiting for replies from the other legislators, the election happened, and the representative from Βοστον was replaced.
The new Βοστον MP was able to learn about all new decrees from a majority <em>before any in that majority received the request to make Λεσλιε cheese inspector.</em>
The representative from Μουνταιν Ϝιεω was away at his villa and did not learn about the election results but did receive the remaining votes for his decree.
At this point, he had received votes from a majority of districts to make Λεσλιε cheese inspector, but it was not true at that moment that a majority of legislators in the current parliament knew about this decree.
Thus, the new MP from Βοστον was able to also make Βαρβαρα cheese inspector (using a majority that did not know about the previous decision).</p>
<h2>The Solution</h2>
<p>At first, the Naxon political scientists blamed the inconsistency on corruption; members of the political elite in Naxos had a habit of taking vacations in Byzantium—to the Naxons, a clear sign of moral bankruptcy.
However, they eventually spotted the problem and came up with a solution.</p>
<p>The Naxon parliament needed a way to distinguish different representatives from the same district.
They decided to refer to each legislator by their district name and the year in which the legislator was first elected (e.g., the gentleman from Σεαττλε, first elected in the year 453).<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> <sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> Then, in all of their messages to each other, the legislators included their current belief about the composition of the parliament.
(I assume that the Naxons had a shorthand way of expressing this, but that particular detail has been lost to history.)</p>
<p>This procedure ensured that the cheese inspector incident could never happen again.
Any legislator attempting to pass a decree while an election was ongoing would learn about any new legislators elected thanks to the added information attached to each parliamentary message (or the decree would complete before the new legislator could become an active member of parliament).
In the example above, the Μουνταιν Ϝιεω representative would have found out that the incumbent from Βοστον had been replaced.
When this happened, under the new rules, the Μουνταιν Ϝιεω MP would have to contact the <em>new</em> representative from the district before considering that district’s vote as counting towards a majority.</p>
<p>There is archaeological evidence to suggest that the Naxons realized this process of assembling majorities and replacing legislators could be used for other civic procedures, ones with weaker requirements than their law-making process, but exactly what procedures they did invent, we cannot say for sure.</p>
<h2>How Paxos Solved the Succession Problem</h2>
<p>On Paxos, thanks to their permanent ledgers and indelible ink, the succession problem was solved much more simply.
A new legislator would be given at the start of their term the ledger of their predecessor and would continue on with the protocol as normal.</p>
<h3>References</h3>
<ol class="acm-bib"><li><span id="lamport98:part-time"><span style="font-variant: small-caps">Lamport, L.</span> The Part-time Parliament. <i>ACM Transactions on Computer Systems</i>. 16(2):133–169. May 1998.</span></li>
<li><span id="recovering-disc17"><span style="font-variant: small-caps">Michael, E., Ports, D.R.K., Sharma, N.K. and Szekeres, A.</span> Recovering Shared Objects Without Stable Storage. In <i>Proceedings of the 31st International Symposium on
Distributed Computing (DISC ’17)</i>. Vienna, Austria. Oct. 2017.</span></li></ol>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>The Naxon parliament could have, instead, referred to each member using their unique names, but they preferred to stand on more ceremony rather than less. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Importantly, any legislator serving non-consecutive terms was referred to by the year of election for their current consecutive tenure. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Fri, 20 Oct 2017 00:00:00 -0700
https://ellismichael.com/blog/2017/10/20/on-ink-shortages/
https://ellismichael.com/blog/2017/10/20/on-ink-shortages/technicalRaft is (Equivalent to) Paxos, VR<p>Warning: This post contains LaTeX rendered with MathJax. It may not render properly in your RSS reader.</p><p>I shouldn’t have to write this blog post, but one too many times have I received
online comments, questions at conferences, and even paper reviews claiming that
Raft is more efficient than Paxos. This is not the case; as Ongaro and
Ousterhout note, <strong>Raft, VR, and Paxos are equivalent in efficiency</strong>.</p>
<p>While the original Paxos protocol was rather concise in its description <a href="#lamport98:part-time">[1]</a>, it can be easily modified into a full-featured state
machine replication implementation. These modifications have been described in
many places (e.g., “Paxos Made Live” <a href="#chandra07:paxo-made">[2]</a>). The normal
case operation of Paxos is visualized and described below:</p>
<p><img src="/img/posts/raft-equivalency/paxos.svg" alt="Paxos Algorithm" /></p>
<ol>
<li>The client sends its command, <script type="math/tex">c</script>, to the current leader.</li>
<li>The leader proposes <script type="math/tex">c</script> in the latest unfilled slot in its log, sending the
proposal to the other replicas.</li>
<li>The replicas put <script type="math/tex">c</script> into the same slot in their logs and respond to the
leader.</li>
<li>The leader waits for a majority of replicas (including itself) to respond and
then executes the command and responds to the client with the result.</li>
</ol>
<p>Viewstamped Replication (VR) was developed at the same time as Paxos <a href="#oki88:view-repl">[3, 4]</a>. While VR includes some practical details
left out of the original Paxos papers, the core mechanism it uses is the same as
Paxos.<sup id="fnref:paxos-vr-diff"><a href="#fn:paxos-vr-diff" class="footnote">1</a></sup> Moreover, the normal case protocol is exactly the same as
described above, and its performance is identical.</p>
<p>Raft was published much later, in an effort to be more understandable than the
existing descriptions of Paxos and VR <a href="#ongaro14:in-sear">[5]</a>, and the Raft
protocol is almost identical to VR.<sup id="fnref:raft-vr-diff"><a href="#fn:raft-vr-diff" class="footnote">2</a></sup> Its normal case operation is
<em>also exactly the same as described above</em>. In terms of message delays, number
of messages sent and received, and amount of computation needed in the normal
case, <strong>Raft has no performance benefits compared to VR or a full-featured Paxos
implementation</strong>.</p>
<h3>References</h3>
<ol class="acm-bib"><li><span id="lamport98:part-time"><span style="font-variant: small-caps">Lamport, L.</span> The Part-time Parliament. <i>ACM Transactions on Computer Systems</i>. 16(2):133–169. May 1998.</span></li>
<li><span id="chandra07:paxo-made"><span style="font-variant: small-caps">Chandra, T.D., Griesemer, R. and Redstone, J.</span> Paxos Made Live: An Engineering Perspective. In <i>Proceedings of the Twenty-sixth Annual ACM Symposium on Principles of Distributed Computing</i>. Portland, Oregon, USA. 2007. 398–407.</span></li>
<li><span id="oki88:view-repl"><span style="font-variant: small-caps">Oki, B.M. and Liskov, B.H.</span> Viewstamped Replication: A New Primary Copy Method to
Support Highly-Available Distributed Systems. In <i>Proceedings of the 7th ACM Symposium on
Principles of Distributed Computing (PODC
’88)</i>. Toronto, Canada. Aug. 1988.</span></li>
<li><span id="liskov12:view-repl"><span style="font-variant: small-caps">Liskov, B. and Cowling, J.</span> <i>Viewstamped Replication Revisited</i>. Technical Report MIT-CSAIL-TR-2012-021. MIT Computer Science and Artificial Intelligence
Laboratory.</span></li>
<li><span id="ongaro14:in-sear"><span style="font-variant: small-caps">Ongaro, D. and Ousterhout, J.</span> In Search of an Understandable Consensus Algorithm. In <i>Proceedings of the 2014 USENIX Annual Technical Conference</i>. Philadelphia, PA. 2014. 305–320.</span></li></ol>
<div class="footnotes">
<ol>
<li id="fn:paxos-vr-diff">
<p>There is a slight difference between VR and Paxos in the way
that invariants are guaranteed across leader elections, but it does not
affect the normal case operations of the protocols. <a href="#fnref:paxos-vr-diff" class="reversefootnote">↩</a></p>
</li>
<li id="fn:raft-vr-diff">
<p>The differences between Raft and VR are detailed in the Raft
paper <a href="#ongaro14:in-sear">[5]</a>. <a href="#fnref:raft-vr-diff" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Tue, 28 Feb 2017 00:00:00 -0800
https://ellismichael.com/blog/2017/02/28/raft-equivalency/
https://ellismichael.com/blog/2017/02/28/raft-equivalency/technicalQuines as a Side-channel<p>I recently competed in <a href="https://icec.tf/">IceCTF</a>, and two of my favorite problems were
about quines. A quine is a program which prints its own source code; the term
was coined by Douglas Hofstadter in his (excellent) book, <em>Göedel, Escher,
Bach</em>.</p>
<h2>The Problem</h2>
<p>The quine problems involved finding a vulnerability in a service (run on a
remote machine) which accepted a C source file as input and repeatedly compiled
the source file, ran the program, and overwrote the source file with the
program’s output. After some number of iterations the server would return the
resulting source code, provided some conditions held (more on that below).</p>
<p>In their purest form, quines should not take any input. For example, the
following Python one-liner is considered “cheating”:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">print</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="n">__file__</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">(),</span> <span class="n">end</span><span class="o">=</span><span class="s">""</span><span class="p">)</span>
</code></pre></div></div>
<p>However, pure quines are quite useless for the purposes of a CTF competition. If
the quine service had only returned one bit of information, whether or not the
input program was a <em>true</em> quine, then it would have been safe from the attack
we used. However, the C programs the server ran could read from files, and the
user the programs were run by had permissions to read <code class="highlighter-rouge">/flag.txt</code>, the file
whose contents we were supposed to exfiltrate.</p>
<p>In the first version of the quine problem, which sent programs through exactly
20 iterations, I had the idea of simply wrapping a target program with layers of
print statements. To illustrate, I’ll do this in Python. Consider the following
target program:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">print</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="s">'/flag.txt'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
</code></pre></div></div>
<p>If we wanted this program to be run during the final iteration and there were 5
iterations, we would wrap it in 4 print statements like so:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">print</span><span class="p">(</span><span class="s">"print(</span><span class="se">\"</span><span class="s">print(</span><span class="se">\\\"</span><span class="s">print(</span><span class="se">\\\\\\\"</span><span class="s">print(open('/flag.txt').read())</span><span class="se">\\\\\\\"</span><span class="s">)</span><span class="se">\\\"</span><span class="s">)</span><span class="se">\"</span><span class="s">)"</span><span class="p">)</span>
</code></pre></div></div>
<p>This approach didn’t work. The service spit out an error, complaining about
source files not matching. We took this to mean that the service would only
print out the final source file <em>if it matched the input source file</em>. In other
words, we assumed that we would only get one bit of information out: <strong>quine or
no quine</strong>. This turned out not to be the case, but I like the solution we came
up with better than the intended one.</p>
<h2>The Attack</h2>
<p>Since we’re assuming that the server only returns the <strong>quine or no quine</strong> bit,
the key insight is that if the programs being run can take some input, then we
can create programs which are quines if and only if some condition holds.</p>
<p>I started with the following C quine, which I found on
<a href="http://stackoverflow.com/questions/10238670/c-c-program-that-prints-its-own-source-code-as-its-output#answer-10240050">Stack Overflow</a>:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include <stdio.h>
</span><span class="kt">char</span><span class="o">*</span><span class="n">s</span><span class="o">=</span><span class="s">"#include <stdio.h>%cchar*s=%c%s%c;%cint main(void){printf(s,10,34,s,34,10,10);}%c"</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">main</span><span class="p">(</span><span class="kt">void</span><span class="p">){</span><span class="n">printf</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">34</span><span class="p">,</span><span class="n">s</span><span class="p">,</span><span class="mi">34</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">);}</span>
</code></pre></div></div>
<p>How does it work? The critical bit is the <code class="highlighter-rouge">%s</code> inside <code class="highlighter-rouge">s</code>. This allows the
program to print <code class="highlighter-rouge">s</code> using a copy of itself. The other trick is that it uses
code points to print out characters that need escaping. (Try representing
newlines as <code class="highlighter-rouge">\n</code> inside <code class="highlighter-rouge">s</code> and removing all the <code class="highlighter-rouge">10</code>s from the <code class="highlighter-rouge">printf</code>
arguments, and you’ll see what goes wrong.)</p>
<p>Using that basic quine as a starting point, I came up with this generator in
Python which creates C quines based on some function, <code class="highlighter-rouge">test</code> (passed to
<code class="highlighter-rouge">gen_quine</code> as a string):</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">TO_REPLACE</span> <span class="o">=</span> <span class="s">"</span><span class="se">\n\r\"\'</span><span class="s">"</span>
<span class="k">def</span> <span class="nf">code_point</span><span class="p">(</span><span class="n">c</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">strip_normal_chars</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<span class="k">return</span> <span class="s">''</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s</span> <span class="k">if</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">TO_REPLACE</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">gen_codepoints</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<span class="n">cps</span> <span class="o">=</span> <span class="s">','</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="n">code_point</span><span class="p">,</span> <span class="n">strip_normal_chars</span><span class="p">(</span><span class="n">s</span><span class="p">)))</span>
<span class="k">if</span> <span class="n">cps</span><span class="p">:</span>
<span class="n">cps</span> <span class="o">+=</span> <span class="s">','</span>
<span class="k">return</span> <span class="n">cps</span>
<span class="k">def</span> <span class="nf">gen_char_s</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
<span class="k">return</span> <span class="s">''</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="s">'</span><span class="si">%</span><span class="s">c'</span> <span class="k">if</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">TO_REPLACE</span> <span class="k">else</span> <span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">gen_quine</span><span class="p">(</span><span class="n">test_function</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span>
<span class="s">"""#include <stdio.h>
#include <stdlib.h>
</span><span class="si">%</span><span class="s">s
char*s="#include <stdio.h></span><span class="si">%%</span><span class="s">c#include <stdlib.h></span><span class="si">%%</span><span class="s">c</span><span class="si">%</span><span class="s">s</span><span class="si">%%</span><span class="s">cchar*s=</span><span class="si">%%</span><span class="s">c</span><span class="si">%%</span><span class="s">s</span><span class="si">%%</span><span class="s">c;</span><span class="si">%%</span><span class="s">cint main(void){if(!test()){exit(1);}printf(s,10,10,</span><span class="si">%</span><span class="s">s10,34,s,34,10,10);}</span><span class="si">%%</span><span class="s">c";
int main(void){if(!test()){exit(1);}printf(s,10,10,</span><span class="si">%</span><span class="s">s10,34,s,34,10,10);}
"""</span>
<span class="p">)</span> <span class="o">%</span> <span class="p">(</span><span class="n">test_function</span><span class="p">,</span>
<span class="n">gen_char_s</span><span class="p">(</span><span class="n">test_function</span><span class="p">),</span>
<span class="n">gen_codepoints</span><span class="p">(</span><span class="n">test_function</span><span class="p">),</span>
<span class="n">gen_codepoints</span><span class="p">(</span><span class="n">test_function</span><span class="p">))</span>
</code></pre></div></div>
<p>For example <code class="highlighter-rouge">gen_quine("int test() {return 1;}")</code> returns:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include <stdio.h>
#include <stdlib.h>
</span><span class="kt">int</span> <span class="nf">test</span><span class="p">()</span> <span class="p">{</span><span class="k">return</span> <span class="mi">1</span><span class="p">;}</span>
<span class="kt">char</span><span class="o">*</span><span class="n">s</span><span class="o">=</span><span class="s">"#include <stdio.h>%c#include <stdlib.h>%cint test() {return 1;}%cchar*s=%c%s%c;%cint main(void){if(!test()){exit(1);}printf(s,10,10,10,34,s,34,10,10);}%c"</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">main</span><span class="p">(</span><span class="kt">void</span><span class="p">){</span><span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="n">test</span><span class="p">()){</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);}</span><span class="n">printf</span><span class="p">(</span><span class="n">s</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">34</span><span class="p">,</span><span class="n">s</span><span class="p">,</span><span class="mi">34</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">10</span><span class="p">);}</span>
</code></pre></div></div>
<p>This is a pure quine.</p>
<p>I’ll leave you to figure out the fine points of the generator. There’s no real
insight, just some fiddling with details. The point is that using it, we can
generate arbitrary (pseudo-)quines! We then used that generator with the
following function:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include <unistd.h>
#include <fcntl.h>
</span><span class="kt">int</span> <span class="nf">test</span><span class="p">()</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">fd</span> <span class="o">=</span> <span class="n">open</span><span class="p">(</span><span class="s">"/flag.txt"</span><span class="p">,</span> <span class="n">O_RDONLY</span><span class="p">);</span>
<span class="kt">char</span> <span class="n">x</span><span class="p">;</span>
<span class="n">pread</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="o">&</span><span class="n">x</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">return</span> <span class="n">x</span> <span class="o">==</span> <span class="sc">'I'</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It worked! The server told us that the generated program was, indeed, a quine.
(An important fact here is that all flags had the format <code class="highlighter-rouge">IceCTF{...}</code>.) From
there, all we had to do was loop through each bit of the flag and test it. Since
all of the flags were less than 80 characters long, this took less than 640
requests to the server (640 separate quines to checks). Not too bad.</p>
<h2>Conclusion</h2>
<p>Was arbitrary (pseudo-)quine generation necessary? Well, no. It turned out that
the server printing out the final source code was important, and the problems
could have been solved in a single request. But our way was more fun.</p>
<p>Moral of the story: if you’re ever building QCaaS (Quine Checking as a Service),
be careful. You might be returning more information than you think.</p>
Fri, 02 Sep 2016 00:00:00 -0700
https://ellismichael.com/blog/2016/09/02/quines-as-a-side-channel/
https://ellismichael.com/blog/2016/09/02/quines-as-a-side-channel/technicalUsing Travis CI with Github Pages<p>I’ve been using <a href="https://pages.github.com/">GitHub Pages</a> for my personal website for some time
now. Since I’ve never needed dynamic content, GitHub has been a perfect
solution. It’s simple to use and free. GitHub will either serve a completely
static site or a <a href="http://jekyllrb.com/">Jekyll</a> site, which it automatically generates.</p>
<p><a href="https://travis-ci.org/">Travis CI</a> is a popular testing and deployment service. You can use it
to generate and deploy your website to GitHub Pages instead of using GitHub’s
automatic Jekyll generator.</p>
<h2>Why use Travis?</h2>
<p>Most of the time, using Travis is not necessary. Vanilla Jekyll works quite well
and has most of the features that you’d want out of a static site generator.
However, GitHub Pages only allows the use of a
<a href="https://help.github.com/articles/using-jekyll-plugins-with-github-pages/">limited number of Jekyll plugins</a> and runs Jekyll builds in
<a href="http://jekyllrb.com/docs/configuration/">safe mode</a>. If you want to use Jekyll plugins other than the ones
available on GitHub Pages, use your own custom plugins, or generate parts of
your site by non-Jekyll means, then you should consider using Travis to
automatically do the job.</p>
<p>My original impetus for switching to Travis was that I wanted the generation
process for my website to generate my resume automatically, which gets built
from a YAML file using LaTeX and a <a href="https://github.com/emichael/emichael.github.io">Python script</a>. However, now
that I’m free from GitHub Pages’ limitations, I’m considering switching from
Jekyll to <a href="http://blog.getpelican.com/">Pelican</a> because I like Jinja templates better than Liquid.</p>
<h2>Making the Switch</h2>
<p>The first thing you need to do is create a new branch, as the generated site
will eventually get put into the GitHub Pages branch (either <code class="highlighter-rouge">gh-pages</code> or
<code class="highlighter-rouge">master</code> depending on the type of repo). Call the new branch <code class="highlighter-rouge">source</code> (or maybe
<code class="highlighter-rouge">pages-source</code> if you prefer).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git checkout <span class="nt">-b</span> <span class="nb">source</span>
</code></pre></div></div>
<p>Next, you’ll want to enable Travis for the repo. You can do so
<a href="https://travis-ci.org/profile">here</a>.</p>
<p>You should then create a file called <code class="highlighter-rouge">.travis.yml</code> in the root of your
repository in the new <code class="highlighter-rouge">source</code> branch.</p>
<h3>Configuring Travis</h3>
<p>Your <code class="highlighter-rouge">.travis.yml</code> file tells Travis what to do. You’ll use it to install your
site’s dependencies, build your site, and then push the generated site back to
GitHub.</p>
<h4>Authenticating Travis with GitHub</h4>
<p>In order for Travis to be able to push your site to GitHub, it will need to
authenticate with GitHub. Travis has built in encryption tools which allow you
to encrypt some secret which only the Travis build agent for your repo will be
able to decrypt. Then, you can upload the encrypted secret to a public repo
without fear that you’re giving everyone access they shouldn’t have.</p>
<p>Some people have suggested using an encrypted GitHub Personal Access Token. This
is a decent solution, but the problem is that the permissions settings for
access tokens are too coarse. They only allow you to determine whether or not
the token has access to <em>all</em> your public repos and <em>all</em> your private repos.
Instead, a better solution is using a deploy key on the repository. If that key
somehow gets compromised, the most anyone will have access to is a single repo.</p>
<p>First, you’ll have to generate the key.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ssh-keygen
</code></pre></div></div>
<p>Name the file <code class="highlighter-rouge">.travisdeploykey</code> and do not put a password on the key. Upload
the public key (in <code class="highlighter-rouge">.travisdeploykey.pub</code>) to your repo (Settings > Deploy
keys). Now, encrypt the private key. You’ll need the Travis client installed to
do that.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gem <span class="nb">install </span>travis
</code></pre></div></div>
<p>Then, encrypt the file.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>travis encrypt-file .travisdeploykey
</code></pre></div></div>
<p>This will output the <code class="highlighter-rouge">.travisdeploykey.enc</code> file and print out a command to be
added to the build script. Save that print out and add <code class="highlighter-rouge">.travisdeploykey.enc</code> to
your repo. Do <em>NOT</em> add the unencrypted private key to the repo. If you already
did, remove the public key from the repo, and start over with a new key. If not,
you can go ahead and delete the private key as well as the public key.</p>
<h4>The Build Script</h4>
<p>Copy the following to <code class="highlighter-rouge">.travis.yml</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">install</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">gem install jekyll</span>
<span class="na">before_script</span><span class="pi">:</span>
<span class="c1"># Replace with the line that travis encrypt-file printed out</span>
<span class="pi">-</span> <span class="s">openssl aes-256-cbc ...</span>
<span class="pi">-</span> <span class="s">chmod go-rwx .travisdeploykey</span>
<span class="pi">-</span> <span class="s">eval `ssh-agent -s`</span>
<span class="pi">-</span> <span class="s">ssh-add .travisdeploykey</span>
<span class="pi">-</span> <span class="s">git config user.name "Travis-CI"</span>
<span class="pi">-</span> <span class="s">git config user.email "noreply@travis-ci.org"</span>
<span class="pi">-</span> <span class="s">COMMIT_MESSAGE="Publishing site on `date "+%Y-%m-%d %H:%M:%S"` from</span>
<span class="s">`git log -n 1 --format='commit %h - %s'`"</span>
<span class="na">script</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">set -e</span>
<span class="pi">-</span> <span class="s">make</span> <span class="c1"># Or jekyll build or whatever command you use to generate your site</span>
<span class="pi">-</span> <span class="s">git checkout -b master</span> <span class="c1"># Replace master with gh-pages if applicable</span>
<span class="pi">-</span> <span class="s">git add -f _site/</span> <span class="c1"># Replace _site/ with the build's output directory</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">git</span><span class="nv"> </span><span class="s">commit</span><span class="nv"> </span><span class="s">-m</span><span class="nv"> </span><span class="s">"${COMMIT_MESSAGE}"'</span>
<span class="pi">-</span> <span class="s">git filter-branch -f --subdirectory-filter _site/</span> <span class="c1"># Again, replace _site/</span>
<span class="c1"># Obviously, replace with your repo's SSH URL and the appropriate branch</span>
<span class="pi">-</span> <span class="s">git push -f YOUR_REPOS_SSH_URL master:master</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="na">only</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">source</span>
</code></pre></div></div>
<p>If you have any other dependencies to install (e.g. Python packages), you should
add the necessary commands under <code class="highlighter-rouge">install</code>. Otherwise, once you make the
appropriate adjustments to <code class="highlighter-rouge">.travis.yml</code>, you should be good to go!</p>
<h2>Final Notes</h2>
<p>Caveat user: debugging your install script and adding all of the dependencies
can be a somewhat frustrating process, especially if the dependencies take a
while to install. You might find yourself making a change, pushing that change
to the repo to kick off a Travis build, and then waiting 5 minutes only to find
out that something else went wrong.</p>
<p>Now that you’re building your site with Travis, you don’t need GitHub to build
it with it’s built-in Jekyll support. By default, GitHub will only invoke Jekyll
if there is a <code class="highlighter-rouge">_config.yml</code> file present, but if you want, you can go one step
further and add a <code class="highlighter-rouge">.nojekyll</code> file to the site. If you’re still using Jekyll,
then you’ll need to add the following to <code class="highlighter-rouge">_config.yml</code>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">include</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">.nojekyll</span>
</code></pre></div></div>
Fri, 12 Jun 2015 00:00:00 -0700
https://ellismichael.com/blog/2015/06/12/using-travis-ci-with-github-pages/
https://ellismichael.com/blog/2015/06/12/using-travis-ci-with-github-pages/technicalSearch For Course Builder<h2>Who I Am</h2>
<p>I am a software engineering intern at Google on the
<a href="https://code.google.com/p/course-builder/">Course Builder</a> team. I attend the <a href="http://utexas.edu/">University of Texas at
Austin</a>, pursuing degrees in computer science and the
<a href="http://www.utexas.edu/cola/progs/plan2/">liberal arts</a>. As a high school student, I made use of free,
online materials, often in the form of college lectures when I was not being
challenged in the classroom. Technology cannot solve all the problems in
education but, I know firsthand that it can help. My decision to work on Course
Builder did not begin with a desire to work in online education, however, but
from my (almost) lifelong love affair with Google, its products, and its mission
to organize the world’s information and make it universally accessible and
useful. When I got a call from my future intern host on the Course Builder team
offering me a summer job at my first-choice company working on a problem I am
passionate about, I accepted the offer rather quickly.</p>
<h2>What Course Builder Is</h2>
<p>Course Builder is an open source project started by Google in 2012 that provides
all of the tools for someone to create an interactive online course. It is
Google’s first foray into a tool of this kind and is already being used by
educators throughout the world. Course Builder is built on
<a href="https://developers.google.com/appengine/">Google App Engine</a> in Python.</p>
<h2>My Project</h2>
<h3>Motivation</h3>
<p>When I was thinking about how I might use my summer, I wanted to build out a
feature that would be immediately useful for students. I thought about various
integrations with other Google products that could facilitate collaboration or
help organize course materials for students, but I kept coming back to one
feature I thought was obviously missing: search! Imagine a student working on a
Latin assignment and struggling to remember how the gerundive is used. In a
traditional course, that student might use the textbook’s index to find the
relative page. In an online course, a student would need to find the relevant
page by searching through the course’s structure manually. If the gerundive was
explained in a video, as content in online courses often is, she would need to
tediously skip through the video a minute at a time hoping to hear something
familiar. It would be much better if the student could simply search for
“gerundive” and be presented with the relevant information.</p>
<p>“But,” you say, “Google already does search! Why reinvent the wheel?” To that I
respond: I’m not trying to reinvent the wheel, but I do not think that a full
internet search is appropriate here. Our Latin student could have searched the
web for information about the gerundive, but she would have been inundated with
complicated and unfamiliar definitions. Moreover, if she were studying a more
arcane subject, she might not find any information at all.</p>
<p>So, I decided to implement search so that any student taking a Course Builder
course could navigate courses quickly.</p>
<h3>Research Phase</h3>
<p>When I stated that I didn’t want to reinvent the wheel, what I meant was that I
never planned on creating a search platform from scratch, or at the very least,
this was a last resort as it would mean that I would have far less time
developing user-facing features. Before I began searching for possible search
platforms, however, I needed a list of key features that I wanted to implement
so that I could assess the relative goodness of each possible solution. I came
up with the following:</p>
<ul>
<li>The platform needed to be fast and support thousands of concurrent queries per
second.</li>
<li>I wanted to be able to search through transcripts of YouTube videos and return
a video fast-forwarded to the relevant spot.</li>
<li>The delay between a change in the course’s content and the corresponding
update of the search index needed to be small.</li>
</ul>
<p>After some searching, I came up with a few options: the <a href="https://pypi.python.org/pypi/Whoosh/">Whoosh package</a>
for Python, <a href="https://www.google.com/cse/">Google’s Custom Search Engine</a>, and the new <a href="https://developers.google.com/appengine/docs/python/search/">Full-Text Search
API</a> for Google App Engine (GAE). Though Custom Search Engine
was initially attractive because it would require a minimal implementation on
the Course Builder side, and the search results would benefit from Google’s
ranking algorithms, I had to rule it out because it would not allow me the
flexibility to search YouTube transcripts. The GAE API was quite new, and I was
wary of it. Whoosh seemed like the most full-featured and stable option, so I
created two quick prototypes, one outside of GAE and one inside it. I subjected
the sans-GAE one to several load tests, and it performed admirably, but when I
tried running Whoosh inside GAE, I ran into problems. Whoosh uses the pickle
module to serialize its search index, and because of differences in the standard
Python environment and GAE, this problem turned into a nonstarter. It may not
sound like a huge issue, but after burning a couple days trying to resolve it, I
found out just how large a problem serialization can be and abandoned Whoosh.
After cobbling together a quick prototype using GAE’s Full-Text Search API
inside Course Builder and subjecting it to a load test, I decided that even
though it was still in the preview stage, it would work and was my best option.</p>
<h3>Design Overview</h3>
<p>If you want to know more about how search engines work in general, you can find
plenty of information via your favorite search engine. If you need help using
your favorite search engine, there’s a great course to help you
<a href="http://www.powersearchingwithgoogle.com/course/ps/course.html">here</a>!</p>
<p>When I decided to take on search, it was immediately obvious that it would fit
in quite well with Course Builder’s existing modular structure. In designing any
peripheral feature for a product, the key is defining and limiting the interface
with the rest of the product and with the feature’s dependencies. In the most
general terms, I decided to separate the search module into two files,
<a href="https://code.google.com/p/course-builder/source/browse/coursebuilder/modules/search/search.py"><code class="highlighter-rouge">search.py</code></a> which depends on App Engine’s search API and contains
all of the module’s interface methods, and <a href="https://code.google.com/p/course-builder/source/browse/coursebuilder/modules/search/resources.py"><code class="highlighter-rouge">resources.py</code></a> which
depends on Course Builder and manages the discovery of course resources. Here,
resource refers to lessons, announcements, external links, YouTube videos linked
to in the course, and others.</p>
<p>I wanted to keep the search interface as small as possible, and I ended up with
three methods: <code class="highlighter-rouge">index_all_docs</code>, <code class="highlighter-rouge">clear_index</code>, and <code class="highlighter-rouge">fetch</code>. These three methods
were sufficient to cover all of the functionality I needed, and limiting myself
to these three helped control the complexity and made the interface easy to
test.</p>
<p>In <code class="highlighter-rouge">resources.py</code>, I needed to handle both resource discovery and differential
presentation in search results (i.e. displaying YouTube videos as videos). In
the interest of clearly separating two concerns, I implemented two classes,
<code class="highlighter-rouge">Resource</code> and <code class="highlighter-rouge">Result</code>, and for each resource in the course that I wanted to
index and search over, I created a subclass of each of these. The subclass of
<code class="highlighter-rouge">Resource</code> is responsible for finding all of the relevant resources in the
course and creating an index document, which is then put into the App Engine
search index. The subclass of <code class="highlighter-rouge">Result</code> is responsible for taking a result
returned by the App Engine search index and formatting it based on the type of
the original document.</p>
<p><img src="/img/posts/search-for-coursebuilder/search_project_overview.png" alt="Search Project Overview" /></p>
<h3>Future Work</h3>
<p>While Course Builder’s search module is a robust, end-to-end feature, there is
still plenty of room for improvement. One thing I would like to see is analytics
for the search history of a course. This would allow course authors to see what
students are searching for most frequently and potentially give them insights
into what students are struggling to comprehend or retain. More resources from a
course, such as questions or PDF files, could be indexed. Also, I think making
the search algorithm context-aware would have great value (e.g. videos and pages
a student has recently viewed are given higher priority in the search results).</p>
<h2>What I Learned</h2>
<p>I do not have the time nor the energy to enumerate all of the things I learned
this summer; I would be writing for days. Instead, I will attempt to restrict
myself to the highlights. Firstly, I learned a great deal about software
engineering and working on a team. For the first time this summer, I went
through a real code review process, and I cannot overstate how valuable it was
to receive almost immediate feedback on every line of code I wrote. My coding
abilities improved, and I began to internalize the reviewer’s voice so that
while I was programming, I was always thinking of what my reviewer might say. I
also learned a great deal about minimizing dependencies and isolating features
so that processes like the indexing pipeline are fault-tolerant and independent
from the rest of the code base.</p>
<p>More importantly, though, I learned a great deal about myself. In a very short
amount of time, having never written production Python code and having never
used Google’s internal tools or infrastructure, I was able to become a
productive member of my team at one of the most technically-proficient companies
in the world. That felt good, and it taught me that I have what it takes to be
able to create good software. I contributed a feature to a product that will
help improve education for thousands of students, and doing that showed me that
I really can affect change in the world. I like that feeling, too, and I think
that for me, this is just the beginning.</p>
<p>So, thank you to Google and the Course Builder team for the opportunity to work
on such a great project and your excellent guidance. It has been a wonderful
summer, and I could not have done it without you.</p>
Wed, 14 Aug 2013 00:00:00 -0700
https://ellismichael.com/blog/2013/08/14/search-for-course-builder/
https://ellismichael.com/blog/2013/08/14/search-for-course-builder/technicalNOPaxos to Appear in OSDIOur paper on eliminating Paxos overhead with network support will appear at
<a href="https://www.usenix.org/conference/osdi16/program">OSDI 2016</a> in
Savannah, GA.
Sun, 31 Jul 2016 00:00:00 -0700
https://ellismichael.com
2016-07-31NOPaxos to Appear in OSDIDiskless Distributed RecoveryOur
<a href="https://www.cs.washington.edu/tr/2016/08/UW-CSE-16-08-02.pdf">paper</a>
on providing stable storage for the Diskless Crash-Recovery failure model is
now available.
Thu, 25 Aug 2016 00:00:00 -0700
https://ellismichael.com
2016-08-25Diskless Distributed RecoveryNOPaxos Madrona Prize Runner-UpNOPaxos was recognized at UW CSE's annual industrial affiliates open house
by <a href="http://www.madrona.com/">Madrona Venture Group</a> for its
start-up potential. See the full post on
<a href="https://news.cs.washington.edu/2016/10/20/madrona-and-uw-cse-recognize-student-innovation/">CSE news</a>.
Wed, 19 Oct 2016 00:00:00 -0700
https://ellismichael.com
2016-10-19NOPaxos Madrona Prize Runner-UpNOPaxos Featured on The Morning PaperNOPaxos was featured on
<a href="https://blog.acolyer.org/2016/12/08/just-say-no-to-paxos-overhead-replacing-consensus-with-network-ordering/">The Morning Paper</a>
blog, with a good write-up of the paper's main contributions.
Thu, 08 Dec 2016 00:00:00 -0800
https://ellismichael.com
2016-12-08NOPaxos Featured on The Morning PaperNOPaxos Recognized at Industry-Academia Partnership Cloud WorkshopMore recognition for NOPaxos. It was awarded the Best Poster Award at the
<a
href="http://www.industry-academia.org/event-university-of-washington-cloud-workshop-2017.html">Industry-Academia
Partnership Cloud Workshop</a> at UW.
Fri, 31 Mar 2017 00:00:00 -0700
https://ellismichael.com
2017-03-31NOPaxos Recognized at Industry-Academia Partnership Cloud WorkshopNOPaxos Source Code AvailableThe <a href="https://github.com/UWSysLab/NOPaxos">source code</a> for
NOPaxos is now available.
Mon, 08 May 2017 00:00:00 -0700
https://ellismichael.com
2017-05-08NOPaxos Source Code AvailableDiskless Recovery at DISC 2017Our paper on diskless distributed recovery has been selected to appear at <a
href="http://www.disc-conference.org/wp/disc2017/accepted-papers/">DISC
2017</a>.
Sat, 01 Jul 2017 00:00:00 -0700
https://ellismichael.com
2017-07-01Diskless Recovery at DISC 2017Eris at SOSP 2017Eris, our network-integrated distributed transaction processing system, will
appear at <a href="https://www.sigops.org/sosp/sosp17/">SOSP 2017</a> in
Shanghai, China.
Fri, 04 Aug 2017 00:00:00 -0700
https://ellismichael.com
2017-08-04Eris at SOSP 2017Eris Source Code AvailableThe <a href="https://github.com/UWSysLab/Eris">source code</a> for Eris, our
network-integrated distributed transaction processing protocol, is now
available.
Wed, 07 Mar 2018 00:00:00 -0800
https://ellismichael.com
2018-03-07Eris Source Code AvailablePaPoC Workshop TalkI recently attended <a href="https://papoc-workshop.github.io/2018/">PaPoC
'18</a> and presented our <a href="https://syslab.cs.washington.edu/papers
/tcdn-papoc18.pdf">short paper</a> on providing efficient causal message
delivery in datacenters using programmable network hardware.
Fri, 27 Apr 2018 00:00:00 -0700
https://ellismichael.com
2018-04-27PaPoC Workshop Talk