A dive into Twisted, SSH, & Git

Our intention was simple — enable people to openly collaborate on projects that could turn into startups. Groups would be open, dynamic, and grow with the popularity of a project.

We needed a place to store our code. The fork and pull model did not meet our needs so integrating with GitHub was not an option. Our proposed access model required branch level permissions. A makeshift Gitolite solution proved to be buggy, difficult to test, and hard to setup.

Building our own git server was the best option. Our main requirement was that it would need to enforce branch level push permissions. Additionally, credentials such as public RSA keys would need to be stored in the database.

Tasked with building the Git server, I quickly found myself knee-deep in Twisted’s source. Several SSH RFCs helped me piece it together. While specifications of Git’s transfer protocols helped guide me towards a suitable solution. What follows is a distillation of what I learned.

Connections — Protocols, Factories, Services

With some Twisted basics we can make the server listen. Protocols are used to manage a connection and receive data. Connections are represented by ITransport implementations.

With a buildProtocol method Factory instances create Protocol instances for each connection. A service is used to bind a Factory instance to a port.

SSHFactory is the base factory class for SSH. It has to be extended to provide the host keys used in the SSH handshake. The Key class represents RSA or DSA keys.

The factory for our Git server ( we call it a “gateway” ) provides its RSA keys.

from twisted.conch.ssh.keys import Key
from twisted.conch.ssh.factory import SSHFactory

from django.conf import settings

SERVER_PRIVATE_KEY = Key.fromFile(settings.GATEWAY_PRIVATE_KEY_FILE_PATH)
SERVER_PUBLIC_KEY = Key.fromFile(settings.GATEWAY_PUBLIC_KEY_FILE_PATH)

class GatewayFactory(SSHFactory):

    def getPrivateKeys(self):
        return {‘ssh-rsa’: SERVER_PRIVATE_KEY}

    def getPublicKeys(self):
        return {‘ssh-rsa’: SERVER_PUBLIC_KEY}

We use a TCPServer service to bind our factory to a port. To start the server as a daemon we put the following in a gateway.tac file :

import sys
import os
os.environ.setdefault(“DJANGO_SETTINGS_MODULE”, “joltem.settings.local”)

from django.conf import settings 
from gateway.libs.factory import factory
from twisted.application import service, internet

application = service.Application(“Gateway”)
internet.TCPServer(settings.GATEWAY_PORT, factory).setServiceParent(application)

And then run twistd :

twistd -y gateway.tac

Authentication — Credentials, Realms, Portals

A user must present credentials when attempting to open up a connection. Twisted stores them in implementations of the ICredentials class.

An implementation of ICredentialsChecker is responsible for verifying the credentials. The requestAvatarId method should return an identification for the user or an UnauthorizedLogin exception. To deal with Twisted’s asynchronous nature, the returned value should be wrapped with the success ( a form of a Deffered ) function, while an exception should be wrapped with the Failure class.

IRealm implementations are responsible for creating an avatar instance representing the connected user. Identification provided by a ICredentialChecker are passed through its requestAvatar method to instantiate the avatar.

A Portal instance is used to attach multiple credential checkers to a realm. Credentials are funneled through a portal, checked against the first capable credentials checker, and if the login is successful create an avatar using the realm.

Since our users provide RSA keys as credentials we use the ISSHPrivateKey implementation of ICredentials in our credential checker.

class GatewayCredentialChecker(object):

    implements(ICredentialsChecker)
    credentialInterfaces = (ISSHPrivateKey, )

    @staticmethod
    def requestAvatarId(credentials):
        … # check source ;)

The requestAvatarId method checks whether it is a valid RSA key. If so computes a fingerprint and uses it to query the database for matching credentials. If successful it returns an instance of our Authentication model representing the RSA key.

The realm then uses an extension of the ConchUser ( SSH specific avatar ) class to create an avatar instance.

class GatewayUser(ConchUser):

     def __init__(self, key):
          ConchUser.__init__(self)
          self.user = key.user
          self.project = key.project

     @staticmethod
     def logout():
          pass

class GatewayRealm(object):

     implements(IRealm)

     @staticmethod
     def requestAvatar(authentication, mind, *interfaces):
         user = GatewayUser(authentication)
         return interfaces and interfaces[0] or None, user, user.logout

Authentication is enabled by attaching our portal to our GatewayFactory instance.

portal = Portal(GatewayRealm())
portal.registerChecker(GatewayCredentialChecker())
factory = GatewayFactory()
factory.portal = portal

SSH — Channels, Sessions, Requests

A small primer on SSH and how Twisted implements it is required before proceeding.

In SSH, requests and data are transferred over channels :

All terminal sessions, forwarded connections, etc., are channels. Either side may open a channel. Multiple channels are multiplexed into a single connection.

To execute something on the remote end a session channel will have to be opened.

A session is a remote execution of a program. The program may be a shell, an application, a system command, or some built-in subsystem. It may or may not have a tty, and may or may not involve X11 forwarding. Multiple sessions can be active simultaneously.

Once the session is opened a request is sent to begin the session’s process. It can either run the default shell, execute a command, or open a subsystem.

Since Git uses SSH to execute a command on the remote end, we are interested in the “exec” request :

 byte     SSH_MSG_CHANNEL_REQUEST
 uint32   recipient channel
 string   “exec”
 boolean  want reply
 string   command

This message will request that the server start the execution of the given command. The ‘command’ string may contain a path. Normal precautions MUST be taken to prevent the execution of unauthorized commands.

Twisted represents channels and sessions with SSHChannel and SSHSession respectively. SSHChannel contains general methods to write and receive data from channels. SSHSession extends SSHChannel and handles session specific channel requests.

Instances of SSHSession contain a session attribute with an instance of an ISession implementing callbacks for processed session requests.

Our ISession implementation is initiated with a GatewayUser avatar an handles “exec” requests.

class GatewaySessionInterface():

    implements(ISession)

    def __init__(self, avatar):
        self.avatar = avatar

    def execCommand(self, protocol, command_string):
        log.msg(“Execute command : %s” % command_string)

We extend the SSHSession class so it will use GatewaySessionInterface as an ISession. There is another way to do this mentioned in the docs, but it is less explicit.

class GatewaySession(SSHSession):

    def channelOpen(self, specificData):
        self.session = GatewaySessionInterface(self.avatar)

To enable our new GatewaySession class we override our avatar’s channel lookup “session” entry.

class GatewayUser(ConchUser):

    def __init__(self, key):
        ConchUser.__init__(self)
        self.user = key.user
        self.project = key.project
        self.channelLookup[‘session’] = GatewaySession

Git — Processes, Reference Discovery, Packfile Negotiation

To serve up Git we will need to understand its protocol and transport mechanisms. Git can utilize several transports, but we are only interested in SSH.

Each time a user fetches or pushes data a command is executed on the server using SSH. The process spawned from the command allows the server and client to communicate.

All commands are handled by two executables on the server : git-upload-pack and git-receive-pack. Sending data to the server fires up the git-receive-pack executable. Fetching data which includes cloning executes git-upload-pack.

For example, the command :

git clone emil@joltem.com:1

Is mapped to this command :

ssh emil@joltem.com “git-upload-pack ‘1'”

Which establishes an SSH connection and executes the command :

git-upload-pack ‘1'

Once the line of communication is opened Git’s protocol proceeds. The protocol uses packet lines ( or pkt-lines ) to communicate :

A pkt-line is a variable length binary string. The first four bytes of the line, the pkt-len, indicates the total length of the line, in hexadecimal. The pkt-len includes the 4 bytes used to contain the length’s hexadecimal representation.

Examples (as C-style strings):

——
 pkt-line actual value
 ————————————————-
 “0006a\n”       “a\n”
 “0005a”         “a”
 “000bfoobar\n”  “foobar\n”
 “0004"          “”
——

A 0000 flush packet indicates an end of a group of statements.

The first order of business is reference discovery :

When the client initially connects the server will immediately respond with a listing of each reference it has (all branches and tags) along with the object name that each reference currently points to.

Along with the first reference in the list the server informs the client of its capabilities.

009bb8915a670ec46e39f25ff700eb260ef5e8d045b1 HEAD\x00multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed
0040cbb40cbc1f82c5d823484b1ccaf742fe210bcf52 refs/heads/develop
003fb8915a670ec46e39f25ff700eb260ef5e8d045b1 refs/heads/master
0040964838da3cd6e95f71d6bf6b425431769f76c22e refs/tags/1.3.2^{}
003d101c5cb5601d9ac04533bf3db810095d0faf7568 refs/tags/1.3.3
003d5da834fffb7e252418b080cca78041517bcd9734 refs/tags/1.3.4
0000

The next step depends on whether the client is fetching or pushing. If the client is fetching a process called pack file negotiation proceeds. During the process the client informs the server what it has and what it wants. The server uses this to compile the necessary objects into the smallest pack file possible.

Since our main goals is to enforce branch level push permissions, let us examine what happens during a push.

Once the client knows what references the server is at, it can send a list of reference update requests. For each reference on the server that it wants to update, it sends a line listing the obj-id currently on the server, the obj-id the client would like to update it to and the name of the reference.

An example, pushing the develop and master branches :

00859d4c1ed658ee2617521d846bf601cec06001a2c8 cbb40cbc1f82c5d823484b1ccaf742fe210bcf52 refs/heads/develop\x00report-status side-band-64k
00676860ded8258f538407e9eb1d2017ffc7a3015abe b8915a670ec46e39f25ff700eb260ef5e8d045b1 refs/heads/master
0000

Immediately after this, the client compiles and sends a pack file. The file contains all the objects the server will need to fill the gap between the old and new commit.

To enforce branch level permissions we decided to create a middleware. Something that sits between the client and server, listens to Git’s transfer protocols, and interrupts when necessary.

Twisted uses ProcessProtocols to connect to and receive data from process. Since an SSH session intends to execute something it is represented as a process. An instance of SSHSessionProcessProtocol is created by SSHSession and passed to our GatewaySessionInterface execCommand method as protocol.

def execCommand(self, protocol, command_string):
    …

The commands the server gets will require execution of git-receive-pack and git-upload-pack in their own separate processes. We extend ProcessProtocols for each of these to mediate the protocol.

Our GitReceivePackProcessProtocol buffers the client’s input and processes packet lines. If a client attempts to push to a prohibited branch the server ignores their pack file and reports back with an informative error message.

To demonstrate our ProcessProtocols requires a lot of code and I didn’t want to complicate the article too much. Luckily, we are an open company and our code is publicly visible. You can clone the repository on our site and test the server out.

If you want to get involved in the development read over this and jump right in.

Caveat — Terminal Access

When a user directly SSHs to our server a terminal appears implemented completely using Twisted. It only handle a few basic commands and I won’t cover it here, but nonetheless some may find it interesting.

Joltem didn’t work out, but the code is open sourced here. Feel free to fork and put it to use.

Building a Git Server was originally published in Joltem — An Open Incubator on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to scale and improve collaboration

Collaboration depends on a compromise between access and control. As a maintainer gives up access, control is lost. With lost control a project may lose conceptual integrity. Tuning access is the foundation of any collaboration.

Our goal is to determine a scalable open source access model. It should allow contributors to easily submit contribution, without prior approval. Scale well beyond ten contributors. As contributors come and go, transience should not produce overhead. Most importantly, communication should be fast and simple.

Fork & Pull

Popularized by GitHub, the most popular access model is the fork and pull. Each contributor not granted read access maintains a copy of the codebase called a “fork”. Work progresses independently on forks. Once completed, a “pull” request is submitted to the maintainers of the codebase to merge in the changes. A technically simple solution that enables mass collaboration. Naturally it has spread quickly despite shortcomings that convolute the collaboration process.

Managing forks and remotes

The most tangible source of friction is the management of remotes and forks. To a setup a contributor must create a fork, clone it, and then add anupstream remote representing the repository forked. Afterwards they must periodically update their clone and fork with changes from upstream. Each contributor represents a remote that must be added and maintained. Fetching changes fires a separate request for each remote. Trivial for small projects, but it fails to scale as the number of contributors grows.

Absence of structure

Branches represent threads that follow a progression. On independent forks, branching structure is defined by the owner of the fork. Context is lost without a consistent naming structure. A multitude of forks erodes meaning even from conventional branches such as a master and develop. Context must be gathered either from commits or by waiting for a pull request.

Communication via pull requests

Communication depends on awareness. Specifically, awareness of the code’s state and direction. When this is represented by a hundreds or even thousands of independent forks it becomes incomprehensible. Sporadic pull requests become the main mode of communication. Combined with a lack of shared branches, communication can become frustrating and slow to a crawl. Code is the preferred communication medium.

Branch Allocation

Working on Joltem we needed an access model capable of housing a dynamic workforce open to contributors. To ease and scale collaboration we chose against the fork and pull model, devised our own scheme, and build a custom git server.

Our model depends on branch allocation. Replacing forks with branches, we enable one repository to serve all contributors. Trusted individuals can obtain complete push access. Other contributors can allocate branches through our task management system. Branch allocation does not require prior approval. It allows the contributor to push changes to the branch.

One remote

The obvious advantage is the single remote. Without setup and maintains of a fork, contributing is simplified. Each contributor no longer represents a separate remote, removing the overhead of maintaing them as contributors come and go. Branch tracking is simplified, making it easier to keep branches up to date. All changes are fetched from a single remote, making it fast and simple.

Definitive structure

Branching structure is well defined. Solution branches are allocated under the s namespace ( i.e. s/123 or s/89). While maintainers of the project decide the rest of the structure. By assigning meaning to branch names they develop and enforce a branching convention throughout the whole project. Convention saves time for everyone involved.

Communication via code

By unifying the codebase, communication is accelerated. The state and direction of the codebase is a git fetch away. The access model provides shared branches between maintainers and contributors, enabling communication directly through code. Small corrections can be made to guide contributors away from compounding on small mistakes.

Joltem didn’t work out, but the code is open sourced here. Feel free to fork and put it to use.

What’s wrong and how we plan to fix it

Late in 2012, I found a cease and desist in my mailbox.

Over four years I had developed and ran a couple products around a popular classified site.

It started in 2008 with an image hosting site that grew quickly and at one point served 2 million unique visitors a month. Then in 2011 an Android app to search and post to the classified site, which has been installed over 200 thousand times.

The whole time I worked alone. I was the system admin, the database administrator, the PHP developer, the Perl developer, the Android developer, the marketing department, the support department, and now the legal department all in one. I did everything from manually deleting nude photos to setting up Bind servers.

As I read the cease and desist which referred to a clause in their terms of use that prohibited development of mobile applications, I reminisced over all the work I had done and became infuriated. However as a lone developer, I had little power and my work would eventually be squashed out of existence. I decided for my next venture I would form a company more resilient to such circumstances.

Building a team was imperative. When working alone you are forced to fill so many holes, that you rarely have the time to do anything properly. In a constant state of compromise which can demotivate and exhaust the most resilient, lone developers are often relegated to chase small ideas with an incredible inefficiency. Overly reliant on other companies, they stand at the bottom of the food chain. In my mind, a serious pursuit would require a team.

As I set out to build a team, I ran across a much bigger problem.

I began by scouring my personal network. Having worked alone for so many years and generally being an introvert, my network was small and contained few developers.

My pool was narrowed to three potential people : a friend who studied computer science, another friend who studied physics, and a friend of a friend who made a website once. We were nearing graduation, so I sat down with my friend from the CS department and pitched to him a partnership. He found the idea compelling but told me he was accepted to a strong master’s program and couldn’t handle the risk. My other friend was in a similar position except with physics. I didn’t bother contacting the third guy.

Maybe it was my fault the pool was so small. Generally I avoided anyone who would ask me what I did. And except for the occasional drunken night, I rarely professed my love of AWS in a public setting. Networking was repulsive to me, but I had to venture out of my comfort zone.

First step, was to set up a profile on MeetUp.com. I searched for groups related to web technologies and entrepreneurship. After a few moments of cringing at the photographs of people meeting each other, I booked two events.

One was an Android group and the other was an HTML5 group. For Android there was about 80 to 100 people and for HTML5 there was about 20 people in the room. In each room there was a couple of intimidating people, a recruiter, and a random group that wondered in for pizza.

After the talks ended, I walked around the room and jumped into some conversations. It was quickly apparent that most people in the room were either beginners or had never programmed. Except for the organizers, there were very few professional programmers in the room. And the few I found were older men who had long lost their inclination to pursue anything ambitious.

I was at a major disadvantage, I was in Dallas, Texas. The concentration of technical talent was small, and it could take me months to find and convince just one other person to partner with. For a second I entertained the thought of moving to San Francisco, joining a startup, going to conferences, and building up network. But I quickly realized how absurd the whole process was.

In an age where people can start revolutions via a tweet, I was going door to door to start a company.

The web is the best distribution system ever devised. It’s cheap, fast, and allows anyone to participate. Yet the companies building it, are formed out of closed, inefficient networks. Outside the network anyone can participate but major disadvantages arise that make many ventures moot. Even for those inside, pursuits often hinge on the willingness of venture capitalist to absorb immense amounts of risk.

The problem was real, I faced it head on and began working on a solution. I imagined a place that people could spontaneously come together and prototype an idea, possibly without any venture capital. A place where anyone in their spare time could participate in forming a company. A place where a completely dynamic workforce could reside.

Over 6 months I worked alone to build an alpha version. At which point I stopped to look for partners again.

Being new to Python and Django, I was constantly landing on StackOverflow threads. I would bookmark profiles of the people whose answers I found helpful. Looking to leverage the internet, this time, I began contacting them one by one. For those that provided contact information, I sent a general email stating I had an interesting project and if they were interested I could discuss it with them over Skype.

Out of around a 100 people, 15 responded, and out of those I Skyped with 10. Some got the idea immediately. Others provided an array of questions, just to have they idea comfortably settle in moments later. During the conversations, it was exhilarating to see the facial expression when the problem was fully highlighted and the realization kicked in. With a caution that I might wrong, I offered a solution and an invitation to peck away at the problem.

Working full time, this took nearly a month and a half. The vast majority did not respond to the initial email. And even though many of the people I Skyped with sounded convinced and mentioned that they wanted to contribute, none actually did. Also making things worse, busy recruiting, I was not coding during the time. It was an incredibly slow and at times demoralizing process. But it hit me, despite the time I had spent only 10 people knew of the idea. A thousand may be necessary just to get one to contribute.

Also the low response rate could be attributed to two things. First, I had no profile. When you Googled my name, buried under my cousin’s LinkedIn profile there was an empty StackOverflow profile where I occasionally answered questions and an even more desolate GitHub profile. No one was listening, and rightfully so. Ironically though I would only discuss the idea with people who had profiles and that would Skype with me. Leading to the second factor, I had fallen into a common trap. I was too protective of the idea. There was little of value built, and I was suffocating it.

Broadcasting the idea was the only way. It required opening up the doors. Revealing the idea and the unfinished code to the world. To seek feedback and contribution publicly.

What came about was Joltem.com, short for “jolt them”. We call it “an open incubator”. Founded on the idea that at least in the early stages many companies can benefit from being open. It was built to organize and compensate an open and dynamic workforce.

We chose against the conventional pull & fork model popularized on GitHub to develop a way for all contributors to work off of one repository. Having everyone work off of one repository facilitates and equips groups for much larger scales of collaboration. Running a simple `git fetch` provides you with all the code from all the contributors. There is no need to add a remote for each contributor and at all times there is only one `develop` or `master` branch to focus development.

To accomplish this we developed a task management model based on tasks and solutions and tied it closely with a custom git server that enforces branch level permissions. Trusted people are able to push to all branches, while all other contributors are only allowed to push to solution branches allocated to them.

For compensation, we simply tweaked a conventional tool — the stock option pool. In regular companies during each round of financing a certain number of stocks is allocated to be awarded to new hires. Hires are then promised a certain number of shares and placed on a vesting plan, where overtime if they stay employed they receive portions of the promised shares. At Joltem, when a company is founded a portion of the shares is allocated towards contributors. For the Joltem project this is 85% of the shares, and essentially represents the conventional stock option pool. The only thing we modified was the distribution of the pool.

Through a continuous process each person’s contribution to the project is tracked by a metric we call impact. Each time a person contributes something to the project they submit it along with the amount of impact they deem fair for the work. The other contributors determine whether the work is satisfactory and whether the evaluation is reasonable. When their is a disagreement, a bargaining process follows to determine a fair value. From experience on the Joltem project itself, there is rarely a disagreement. A contributor quickly develops a sense of what each contribution is worth. Then periodically, much like a vesting plan, a portion of earned impact is exchanged for shares in the company that is being built.

Now, Joltem, is open to anyone.

You can post a project. Collaborate privately by inviting others or completely openly. You can even contribute to the Joltem project itself.

Joltem didn’t work out, but the code is open sourced here. Feel free to fork and put it to use.