Today it has been 5 years since I & Sneha Jain joined Krishnan Nair in building Geektrust.

Well of course, it had been one of the hardest decisions in my life to start up, but 5 years down the lane there are absolutely no regrets. It has been incredible working with Krishnan and Sneha.

From the days of roaming around and working from the cafes of Koramangala and Indiranagar, we have slowly graduated to 1 BHK & 2 BHK offices and slowly and steadily built a team of 12 which include tech, ops & sales. We could remain lean because of the hell lot of automation that went into Geektrust. And most importantly we do exist without getting a penny of VC money. Yes! We exist :D

These 5 years were not just of learning technically, but also about learning how a company is built and a product is sold. I had always thought that building a product is the hardest part, and always said it with pride. Today, with the 2 products we built in 5 years, I would say building/developing is easy-peasy. The hardest thing is selling the product. And mind you, this is coming from a person who codes 5 days a week for the last 17+ years.

Personally, the biggest challenge I faced in these 5 years is building the code evaluation automation engine, which we have aptly named Codu (codu.ai), as a tribute to code and Bengaluru. Codu today can predict Readability & Object Orientedness of your program, based on the code evaluation we have done in the past.

Thanks to a lot of people who helped us in all parts of our life with Geektrust. And to those amongst us.Without the team of Clair, Midhu, Luka, Meenu, Shiby, Akshay, Adarsh, Gaurav & Purnima we couldn’t have built Geektrust as it is today. Thanks to Madhav for being a guardian angel.Thanks to all our clients who trusted us and hired through us, and released the cheques faster to stay afloat.

Last, but not the least, thanks to the community of software developers who took our coding challenges and trusted us with the opportunities they got through us. Without your help and word of mouth marketing, we couldn’t have survived 5 years.

Thank you all for making my last 5 years beautiful.

Codu is an ML product, built by us, that reviews code how a developer would, for parameters that we’ve seen mostly matter in the real world, which is the ability to write clean code — code that others can read and understand.

You can read about why and how we built it.

At Geektrust, we use RabbitMQ as our message broker for communicating between distributed applications.

Recently when we introduced RabbitMQ in Codu.ai, we started seeing this issue of the connection getting dropped when the publisher/consumer is inactive for a long period of time.

Codu is built using Go, and uses the Go client supported by RabbitMQ. It is currently in private beta, and there are only a few regular users. Once in a while, when someone sends a request, they don’t get any response because the Go client has lost its connection with RabbitMQ server. As the client library doesn’t give an option for automatic connection recovery, we had to implement it ourselves.

We have only one publisher process and one consumer process that will publish and consume the messages to and from multiple queues in the same exchange. The consumer listens to messages from multiple queues and handles these messages differently based on the message type. Most of the sample code we saw, showed the example of one consumer for one queue, which was slightly different from our implementation. In this post, we show how we handled the auto-reconnect of a consumer and publisher even when connected to multiple queues.

Setting things up

The above gist shows how a RabbitMQ connection is established. I use a custom Connection struct to encapsulate all the necessary information. An object of this struct is returned by using the function NewConnection on the comms package. All the connections can be managed in the pool against a name, if you have multiple consumers and publishers. After creating a Connection object we need to call the Connect and BindQueue methods, so that the connection is established and queues are bound to the exchange. There is also a Reconnect method which is called when the connection is lost.

Now coming to the most important part. In the Connect method, a Goroutine is called, which will wait on after registering a listener on the NotifyClose method on the amqp.Connection object.

From the AMQP library docs:

NotifyClose method on the amqp.Channel object registers a listener for when the server sends a channel or connection exception in the form of a Connection.Close or Channel.Close method

When the ‘connection close’ event is received, we push a custom error to the err channel of the Connection object.

go func() {
     //Listen to NotifyClose  
     <-c.conn.NotifyClose(make(chan *amqp.Error)) 
     c.err <- errors.New("Connection Closed") 
}()

You need to be waiting on this channel and if there is an error, we call the Reconnect method, where we reestablish the connection. We will see how this is used in the consumer and publisher.

The Consumer side of things

In this gist you can see theConsume method which will loop through the queues and consume the messages from the amqp.Delivery channel. Once consumed, it is stored in map (this is not thread safe) and returned.

The HandleConsumedDeliveries method will handle these messages that are consumed from the channel for each queue in an infinite loop. It also takes in a function of definition func(Connection, string, <-chan amqp.Delivery) which is the actual function that handles your message. This will be called in a separate go routine so that the messages are handled without any blocking. In this function we also wait for the err channel on the Connection object.

if err := <-c.err; err != nil {   
     c.Reconnect()   
     deliveries, err := c.Consume()   
     if err != nil {    
        panic(err)
     }
     delivery = deliveries[q]  
}

On the NotifyClose listener defined while initialising the connection an error object is pushed to this channel, whenever the RabbitMQ channel/connection is closed. Here, when the error obtained is not nil, we reconnect the connection, and start consuming the deliveries again. The delivery channel for the specific object is set again so that the messages are picked up again and the fn object is called again. The consumer section below shows how all this is wired together.

The Publisher side of things

The publisher side of things is relatively easy.

Here, we use a non-blocking channel setup with the select keyword, as there are no Goroutines calling this method. So, if there is no error on the err channel, it just goes and publishes the message. Else if the connection is lost, it will reconnect and then publish the message.

The Consumer application

The consumer main program is waiting forever as it has to wait for the messages arriving in the queues. We create a new Connection object, and call the methods Connect and BindQueue on it. Then, we consume the queued messages using the Consume method. Once this is done, we loop through all the queues, and call the HandleConsumedDeliveries method on each of them in a separate Goroutine. Notice that we pass a function messageHandler of type func(Connection, string, <-chan amqp.Delivery), which actually deals with the raw message obtained from the queue. Here I am only printing a message using it.

The Publisher

In this application, we create a new Connection object and then call the methods Connect and BindQueue on it. We loop through the queues and publish the message into each of this.

This program will be executed one time, so it won’t have the ‘connection close’ issue. But if you are running a web application and pushing messages to RabbitMQ, there is a chance that the connection will be closed on too much idling. At that time, the auto reconnect feature will come in handy.

Hope this was useful for some of you folks, and let me know if you got any questions in the comments section.

The high level idea is to get the candidate’s code inside a docker container and execute it. This docker container is created for the programming language she has solved the problem in. When the container is created an executable will be run to test the input and output of the candidate’s code. The executable will run the candidate’s code using the language specific execution command. For e.g. for Python it can be python main.py

For this we have to build Docker Images. These images will have the execution environment for each language, that a candidate will submit the code in. So there will be a Docker image for Java 1.8, another for Java 1.11, one for Ruby 2 etc. These docker files are pre-defined. But the code that executes the candidates code, inside the container, is not available as part of it. This is because it is bundled as a Go executable file. We do not want to build that every time and check it in along with the Dockerfile into the git repo.

So instead of doing a docker build -t build_name . we build the image programmatically using the Go client for Docker’s Engine API.

Docker Build Context

Before we get into building the image programmatically it is important to understand what a Docker build context is.

Docker build context is essentially the source directory from where you execute your Docker builds. For the given Dockerfile

the source directory will look like this

.
├── Dockerfile
├── test.env
└── test.exe

This Dockerfile will build an image from Alpine Linux base image. When run, a program test.exe will be executed inside the container.

The step COPY ./test.exe ./ copies the executable file test.exe from the host machine to the work directory of the Docker image that is built. All the files you need have inside the container should be in the Docker build context before you build the image.

Building the Image

There are different ways to build a Docker image. The traditional way is to build from the source directory (docker build -t tag.) We can also build an image from a TAR file. In this case we pass the TAR file which contains the Docker build context to the docker build command. We build the Docker image programmatically this way.

The steps to build the image in our case are the following.

1. Clone the repo which has the Dockerfile to a temporary folder.
2. Download & copy the pre-built exe to the temporary folder created from the above step.
3. Create the tar file of the Docker build context from the temporary folder.
4. Build the image using Docker Engine API using the tar file we created.

The code for step 4 is given below

In the above code BuildImage takes in the Docker source directory location and a tagName. A context instance is created with 5 minutes timeout. dockerFileTarReader, err := tarDirectory(dockerBuildCtxDir) will convert the directory into a Tar file of type io.Reader . You can use Matt Holt’s archiver in Go for creating a tar. Once the tar file is created we pass it onto the ImageBuild api of the Docker client instance. This client instance can be created from the Go client library for the Docker Engine API

resp, err := cli.ImageBuild(
  ctx,
  dockerFileTarReader,
  types.ImageBuildOptions{
   Dockerfile: "Dockerfile",
   Tags:       []string{tagName},
   NoCache:    true,
   Remove:     true,
   BuildArgs:  buildArgs,
  })//cli is the docker client instance created from the engine-api

The ImageBuildOptions take the options required for the Docker build. It is the Go struct for the build options in the docker build command. NoCache ensures that cache is ignored when we run the Docker build every time. Tags adds the tags you want to give to the build. Remove ensures that the intermediate containers created are removed after a successful build.

The response returned has a Body of type io.ReadCloser which can then be read for the logs. These logs are exactly the same when you run a docker build from the command line.

Once the call to BuildImage successfully returns, we have built a docker image programmatically.

You can plug this code where ever you want to build such docker images on the fly. You can verify the availability of the image using the docker images command on the host machine. You can also do a docker run <image_name> to create and execute the container.

[About the author — Dhanush is co-founder & CTO at Geektrust.

Geektrust is built for technologists to connect with remarkable job opportunities.]

Join our community Slack and read our weekly Faun topics ⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

How to build Docker images on the Go was originally published in FAUN.dev() 🐾 on Medium, where people are continuing the conversation by highlighting and responding to this story.

I had the opportunity to interview Minister G Sudharakaran at his office. While talking with him in detail, Kondotty MLA and Muslim League leader Muhamadunni Haji came in with an old lady. She came with the MLA to see the Minister for requesting something.

Some years back, her son had taken a loan from a nearby co-operative bank, by pledging their house, and left the town without paying it back. Now the interest and the principal amount has gone up and the bank decided to attach the property. The poor old lady has no where to go.

Her life, explained in tears, moved the minister. He forgot the interview and involved in fixing her issue. We all know how difficult are the bank formalities. But that day, I realised a minister can remove any red-tapes in governance, if he/she takes it up.

10 phone calls. Strict instructions in his usual style .. that was enough. In 30 min he included the old lady in some scheme and relieved her of all the issues.

With teary eyes when she thanked the minister and left, he said to me. “Poor. The MLA is a good soul, so he brought her. Otherwise her property would have gone”.

How much ever we criticise the Kerala politicians, in between all the politics they play, they do a lot of good for the common people. When bureaucracy, citing laws, fails to provide support to the poor and needy they can go to only place and knock. That is their representative in the state assembly. Whichever Kerala politician’s house you go, you can see people standing there with their painful stories. There still remains some goodness in the name of humanity amongst our politicians, irrespective of the party and flag they represent. That is why a Ramesh Chennithala and Pinarayi Vijayan can travel together in the same helicopter and reach the places where calamity has struck.

The point I am coming to is that, whoever is ruling, one of the good thing about Kerala is its Chief Minister’s Distress Relief Fund. Irrespective of the party lines deserving people get it. Kerala is one of the states of India, which distributes the CMDRF in an efficient way.

Last UDF government disbursed 800 Crore Rupees to the needy in the its 5 years. This Pinarayi government have disbursed 423 crores in the last 2 years. It has helped 234899 people stay afloat. (Donations to CMDRF is less. Most of it comes from the Lottery sale)

This government made the process easier. The disbursement is via DMR software these days. Now you need not go looking for the MLA, need not go to Thiruvananthapuram. Everything can be done online. You can even get the status of it online. Once applied the money will be in the account within 100 hours. When this government took charge there was 30000 requests waiting for them. That was all settled very fast. The government gave permissions to Thahasildars to disburse the money, after opening an account in the state treasury.

If you want to know the goodness of Kerala in this, you need to look at other states. In Maharashtra brokers made fake documents to get the amount. In Karnataka, the fund itself was converted into an FD by the officials, in 2010. Goa spent it in extravaganza and Punjab brokers took it.

In Kerala there had not been a single instance of complaint against the CMRDF. The fund is distributed to people who have serious illness, people who have met with accidents and to people who have met with natural calamities. The District Collector, Revenue special secretary and Revenue minister can disburse upto 10000, 15000 & 25000 Rs respectively from this fund. Chief Minister can disburse upto 3 Lakhs rupees. Above that you need the permission of the senate/assembly.

In this times of distress and calamity our CM, Shri Pinarayi Vijayan has been requesting time and again to individuals and companies to donate to this fund. In reality the chief minister is extending his hand of request for our state. We can be proud that every rupee that reaches the CMDRF is distributed to the people who deserve it. From July 26th to August 9th 2018, 1.75 Crore rupees has been reached this fund . You can send your donations to this account

Bank : State Bank of India (SBI)
Account Number : 67319948232
Branch : City Branch, Thiruvananthapuram
IFSC : SBIN0070028
PAN: AAAGD0584M
Account Type: Savings
SWIFT CODE : SBININBBT08

UPI: keralacmdrf@sbi
Website: https://donation.cmdrf.kerala.gov.in/

My state, Kerala, has been reeling under heavy monsoon and floods. A lot of people have lost their lives and belongings. According to reports, so far 94 people have lost lives and more than a lakh of people are staying in relief camps and other places. The state government is in full action and helping everyone to evacuate and rescue. Central government is providing the necessary support. But we need your help. We are in need of funds and supplies.

An organisation Anbodu Kochi has setup different collection centers in Bangalore to collect supplies. You can donate to the Kerala CM’s Distress Relief Fund through here. You can be sure that it will reach the poor & needy as Kerala is one of the best states that manages its CMDRF fund well.

You can do NEFT transfer to this account.

Bank : State Bank of India (SBI)

Account Number : 67319948232

Branch : City Branch, Thiruvananthapuram

IFSC : SBIN0070028

PAN: AAAGD0584M

Account Type: Savings

SWIFT CODE : SBININBBT08

Or do an UPI transfer to keralacmdrf@sbi

Following are the places in Bangalore, where Anbodu Kochi has setup collection centers.

And they are in need of these items

Please do share this message with your other groups.

I am sure our God’s own country will be back to its glory days soon with all your help.

Go has an http package which has simple APIs to do HTTP operations. For e.g. if you want to download a page all you need to do is this.

resp, err := http.Get("http://example.com/")
if err != nil {
	// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)

Given below is the implementation of downloading a file from a public URL and using it as bytes.

For downloading multiple files concurrently you can use the help of goroutines and channels.

In the above code snippet, we initialise two channels, one for sending the file downloaded as []byte type and another for sending any errors that happened while downloading.

We then loop though all the urls and call the downloadFile method for each of them in a goroutine. These goroutines are executed concurrently and so the for loop is finished processing in no time. Inside the goroutine each file is downloaded and we get back bytes and error.

If there is an error in the download, then the error is pushed to the error channel and nil value is pushed to the done channel. If there is no error then the bytes are pushed to the done channel and a nil entry is pushed to the error channel.

An array of byte arrays is declared to collect the downloaded files back. We again loop through as many times as there are URLs to collect the byte array back from the done channel.

bytesArray = append(bytesArray, <-done)  
if err := <-errch; err != nil {   
   errStr = errStr + " " + err.Error()  
}

These two lines inside the for loop wait for the done and the error channel to receive data. Once the data has arrived, it increments the loop count. Bytes are appended to the array of bytes while error string is appended, if there are any errors.

You have now downloaded the files concurrently and they are available as an array of bytes. All just using simple Go code.

Sometime back at Geektrust, we were up with our recommendation engine, recommending kick-ass developers to our clients based on their profile, years of experience and technology. It was built on Go & Elasticsearch. What we did was match the user profiles for the requirements of the companies. In addition we matched the best developers who had a close similarity to what the company was looking for. We took small steps into the world of Machine Learning at that time by doing similarity matches.

But, we had a problem. We were relying on the technologies that our users entered, and people always entered Java. Not everyone, but at least 60–70% of them did that. This resulted in QA Engineers with some Java expertise getting recommended to a company that was looking for a hardcore Java Backend developer. This didn’t look good in front of the customer and we decided to tighten up our recommendations. What we needed to do was identify what kind of developers were registering on our site. And that’s where Machine Learning came to our help.

Training Data

We went with supervised learning, which means classifying data based on previously labelled training data.

As part of this, the first thing we did was to list down the features we wanted to identify in a developer to classify him/her as a Backend or Frontend or Fullstack developer. Keeping that in mind we came up with a set of features like count of backend technologies in a user and scan their resumes to understand what kind of technologies have they worked on.

Once these were identified, we churned out a file that marked these features against each developer in our database. This was the first step in creating our training data.

In the above file, the columns indicate the features and the last column indicates the item we want to predict. In the training data the existing data was cleaned and marked with a specific role. This can be generated with scripts, but without 100% accuracy. We had to modify our training data to label the column that needs to be predicted, which is role. 6 of us worked through around 7000 rows of data and verified, validated and modified the data, if required.

Training & Testing

Once the training data was ready we had to test the accuracy of this model. We tried different classifiers like K Nearest Neighbours (KNN), Decision Trees and Support Vector Machines(SVM). We also had to go back and fine tune the training data, adding more features, removing some of them. All these decisions were based on the accuracy we got on these different classifiers. SVM gave us the best accuracy and we went ahead with it using a linear Kernel. The implementation was in Python. With well established libraries like scikit-learn , NumPy, SciPy and pandas available, we had little doubt in selecting Python as our go-to language for ML.

The above code shows how you can train & test your model. The trained data in csv file is split into test & train data and then the test data is used to check if the prediction is correct or not. It also gives you the probability percentage of the prediction. The predictions made by the SVM were verified against the already trained data and it was pretty accurate.

And that’s it. Now for every profile update the developer does on Geektrust, we build the feature data and predict the role from it. Once we predict what kind of a developer he/she is, we use that information to recommend them accurately to a specific requirement of a company. Now our recommendations to companies are better than earlier.

However, this doesn’t guarantee that the predicted role is correct all the time. Sometimes, we get predicted role with probabilities as low as 20–30%. In such cases we manually override the prediction, to what we think is the role of the user. We also update the existing training data at periodic intervals, which helps in improving future predictions.

At Geektrust, we continue to work on ML & AI related stuff and this is an ongoing activity. We have started working on our own resume parser and look forward to building an advanced code evaluation engine that checks for clean code.

[About the author — Dhanush is co-founder at Geektrust and takes care of all things tech.

Geektrust is built for technologists to connect with remarkable job opportunities.]

The idea was to execute the code in the isolated container and log the output/error in JSON format, and then read this log to send the response back to the developer who is executing his/her code. A REST service will receive the developers code and language in which he/she has written. This service is written in Go and I used Docker’s docker-engine API for creating & running the container. We read the logs using the same API.

However there is a catch while reading the logs from stdout stream of container. The data written by the container is in a format which has a HEADER & PAYLOAD. The header contains the information about the stream and payload and is encoded on the first 8 bytes like this.

header := [8]byte{STREAM_TYPE, 0, 0, 0, SIZE1, SIZE2, SIZE3, SIZE4}

If you read the container logs like this

then you will see unicode characters printed on the console and then the actual payload. If you expect a JSON from the console and is ready to decode the JSON to a struct like this

json.NewDecoder(strings.NewReader(string(content))).Decode(&myStruct);

then this will fail, because it is reading data which contains unicode characters and JSON.

So we have to skip the first eight bytes of the stream and read from the reader like this.

When I wrote the code first (like in the first gist), I was not aware of this issue and I couldn’t get what these unicode characters were. The documentation on the remote api for getting the container logs doesn’t have a mention about it. It was this issue raised on Github that pointed me to docker remote-api’s Attach Container documentation which explains about HEADER & PAYLOAD. Hopefully some body will update the remote API’s documentation.

The code is as given below.

In this I encapsulate the smtp request in a struct Request. It contains basic things like To, Subject, Body. This template file has the data placeholders which will be replaced with actual data values by Golang’s html/template package Execute method.

The template data is a struct which has Name & Url as the field values. It is initialised with values and passed to the NewRequest method to create an object of Request.

templateData := struct {
    Name string
    URL string
 }{
    Name: “Dhanush”,
    URL: “http://geektrust.in",
 }

Once the instance of Request is created, then ParseTemplate method is called on it. This method receives a template filename and template data. It parses the template file and executes it with the template data supplied. A bytes.Buffer is passed to the Execute method as io.Writer so that we get back the HTML string with the data replaced. This HTML string is then set as the Email Body.

The SendEmail method sets the MIME encoding as text/html and calls smtp package’s SendMail method to send the email. When the email is sent successfully the SendEmail method returns a true value.

Note:

1. If you return the Request in ParseTemplate method, instead of the error, then we make the call in one line.

ok,err := NewRequest([]string{“junk@junk.com”}, “Hello Junk!”, “Hello, World!”, “template.html”, templateData).ParseTemplate().SendEmail()

But then any error coming out of ParseTemplate may have to be tracked inside the method.

2. Read the package documentation of html/template to understand how Golang replaces the HTML with actual data.

3. Thanks to Jijesh Mohan and Navaneeth K. N in making the Go code more idiomatic, than it was before :)

GeekTrust, from its start last year, has always been a Javascript shop. I selected Javascript as at that point of time, because we wanted to go to production quickly. We had to try out our business idea super fast. And Javascript helped us do that.

I have been coding Java since 2003 and always been a fan of it. I would also admit that it solved a lot of complex problems for me. In August 2015 I just waved goodbye to Java. And I picked up Javascript actively (I have done some JS coding before on the job).

Around 3 years back, I decided that I wanted to be a Polyglot programmer. I wanted to learn something new in programming. I had shortlisted Python, Go, Haskell and Javascript as the languages I wanted to be an expert on. I went and did an Udacity course in Python. Learned Go at the free time and kept reading about Haskell. But I didn’t become a good programmer in any of those except Python.

Last year I attended GopherCon India and was mesmerised by the talks various people gave. I had to learn and write code. I kept doing simple things on my github. I even extended William Kennedy’s Ping pong problem to Cricket, my favourite sport.

When we started GeekTrust I wanted to write everything in Go. But I was still not fluent in Go, and I knew it will take time to learn, so I chose JS. This year too I attended GopherCon India. After that keynote from Mark Bates, I realised I am also fighting the Go FUD. If you have fear, uncertainty and doubt about Go, you should watch that talk. And around that time, we had made some decisions about our product roadmap. I saw an opportunity to build an independent service and I decided to develop it in Go. As I started developing, I realised that I can do things fast and it was quite simple to do it. I was eliminating FUD.

Matthew Campbell’s talk and Bryan Liles’ article helped me understand about monorepo. The recommendation engine wouldn’t have seen daylight without Oliver Eilhard’s beautiful Golang client for Elastic. Many thanks for that Oliver. I could easily convert the Elastic JSON requests to code and would highly recommend it. Jijesh Mohan helped a lot with insights into code organisation, monorepos, middleware & what not. And so has the Golang communities.

At this moment, we are on the path to do new things in Go. Soon, I hope, the entire GeekTrust platform would be Powered by Go.