LLMs are everywhere now. We are building different applications utilizing them to make our lives easier. However, accuracy is not always guaranteed; models can hallucinate and may struggle with providing quantitative responses. While many reports evaluate the models themselves, those results don’t always translate well to our own applications. So, what should we consider when evaluating an LLM specific to our application?

When evaluating a particular application or task, I like to think of a combination of prompt, model, and configurations as the input to the system. Adjusting any of these-like tweaking the model’s temperature-changes the output. Even removing one or two words from the prompt can alter the outcome. So, let’s consider all these factors as part of the input, understanding that any adjustments mean we’re changing the input itself.

Throughout this article, I’ll use the example of categorizing customer support tickets to explain everything, but these methods are applicable to any use case. Let’s dive in.

Garbage In, Garbage Out

Before we even start measuring the accuracy of our application, we need a good dataset. This should be a list of example inputs and expected outputs that are accurate for your domain, application, or industry. Here’s an example:

+----+------------------------------------------------------------------------------------+-------------------+
| ID |                               Support Ticket                                       |     Category      |
+----+------------------------------------------------------------------------------------+-------------------+
|  1 | "I forgot my password and can't log in. Can you help me reset it?"                 | Account Management|
|  2 | "My credit card was charged twice for the same order. How do I get a refund?"      | Billing           |
|  3 | "The website keeps giving me an error when I try to submit my order. Can you help?"| Tech Support      |
|  4 | "I received a large instead of a medium size shirt. How can I exchange it?"        | Product Support   |
|  5 | "I'm having trouble accessing my account."                                         | Tech Support      |
+----+------------------------------------------------------------------------------------+-------------------+

This looks like a solid set of inputs, right? But let’s take a closer look at the last one:

“I’m having trouble accessing my account.”

Depending on your company, this could fall under “Tech Support” or “Account Management”. Without more context, it’s hard to say. You need to ensure the categorization aligns with your company’s policies, workflows, and expected behaviour. How would a person triaging these tickets classify this one? For our fictional company, they would prefer to categorize this ticket under “Account Management” since it’s more likely to be a password issue than a technical one.

This is why we need to ensure that our inputs are solid before we start measuring accuracy.

Metrics

Now that we have a good dataset as a foundation, how do we actually measure accuracy? There are many metrics we can use. Here are some common ones that I rely on.

Basic Accuracy

This metric is generally used when you have a right or wrong answer. For our example, it would be whether or not the category output is correct. When it is correct, we give it a score of 1; when it is incorrect, we assign a score of 0. Here’s a sample:

+--------------------+-------------------+-------+
|     Expected       |    LLM Output     | Score |
+--------------------+-------------------+-------+
| Account Management | Account Management|   1   |
| Account Management |   Tech Support    |   0   |
|      Billing       |  Product Support  |   0   |
|    Tech Support    |   Tech Support    |   1   |
+--------------------+-------------------+-------+

Based on this, the total accuracy would be 50% as it got 2 out of 4 correct.

F1 Score

What if we expected a list of categories for each support ticket? For example, we might expect ambiguous tickets to be categorized under multiple categories. Instead of choosing one or the other, the model could specify both. Now, it’s no longer a simple right or wrong scenario. Here’s an example output:

+----+----------------------------------+------------------------------------+
| ID | Expected                         | LLM Output                         |
+----+----------------------------------+------------------------------------+
|  1 | Account Management               | Account Management                 |
|  2 | Account Management, Tech Support | Tech Support, Billing              |
|  3 | Billing                          | Product Support                    |
|  4 | Product Support, Billing         | Product Support, Account Management|
+----+----------------------------------+------------------------------------+

To measure this, I find that the F Score is the best approach, as it considers false positives/negatives as well as true positives/negatives. Here’s the formula:

Precision = True Positives / (True Positives + False Positives)
Recall = True Positives / (True Positives + False Negatives)

F1 = 2 * (Precision * Recall) / (Precision + Recall)

Let’s calculate it for ID-2:

True Positives = 1 (Tech Support)
False Positives = 1 (Billing)
False Negatives = 1 (Account Management)

Precision = 1 / (1 + 1) = 1/2
Recall    = 1 / (1 + 1) = 1/2

F1 = 2 * (1/2 * 1/2) / (1/2 + 1/2) = 2 * 1/4 / 1 = 1/2

Doing this across the entire set:

+----+----------------------------------+------------------------------------+-----------+--------+----------+
| ID | Expected                         | LLM Output                         | Precision | Recall | F1 Score |
+----+----------------------------------+------------------------------------+-----------+--------+----------+
|  1 | Account Management               | Account Management                 |       1.0 |    1.0 |      1.0 |
|  2 | Account Management, Tech Support | Tech Support, Billing              |       0.5 |    0.5 |      0.5 |
|  3 | Billing                          | Product Support                    |       0.0 |    0.0 |      0.0 |
|  4 | Product Support, Billing         | Product Support, Account Management|       0.5 |    0.5 |      0.5 |
+----+----------------------------------+------------------------------------+-----------+--------+----------+

This gives us an overall accuracy of 50%.

Mean Squared Error

If we expect the output to be on a scale or rating system, we might want to use a metric that considers how close the output was to the actual answer. Let’s assign these tickets a priority between 1 and 3, with 1 being low priority and 3 being high priority. We want to account for the fact that if we expected a rating of 3 and the output was 1, that should be considered less accurate than if the output was 2. Here’s an example:

+----+---------------------+------------+
| ID | Expected            | LLM Output |
+----+---------------------+------------+
|  1 | 2 - Medium Priority | 3          |
|  2 | 3 - High Priority   | 2          |
|  3 | 3 - High Priority   | 1          |
|  4 | 1 - Low Priority    | 1          |
+----+---------------------+------------+

The metric to use here would be the Mean Squared Error, which measures the amount of error in the output. We can then invert this to get the overall accuracy. Here’s the formula:

MSE = (1/n) * Σ(actual - predicted)²

where:
- n is the number of data points.
- actual is the true value.
- predicted is the predicted value.

Accuracy = 1 - (MSE / Max Squared Error)

Let’s calculate the squared error for the first row:

SE = (actual - predicted)² = (2 - 3)² = (-1)² = 1

Doing it for the rest:

+----+---------------------+------------+---------------+
| ID | Expected            | LLM Output | Squared Error |
+----+---------------------+------------+---------------+
|  1 | 2 - Medium Priority | 3          | 1             |
|  2 | 3 - High Priority   | 2          | 1             |
|  3 | 3 - High Priority   | 1          | 4             |
|  4 | 1 - Low Priority    | 1          | 0             |
+----+---------------------+------------+---------------+

Now, let’s calculate the rest, noting that the Maximum Squared Error would be (3 - 1)² = 4 based on our scale from 1 to 3.

MSE = (1/n) * Σ(actual - predicted)² = (1/4) * (1 + 1 + 4 + 0) = 1/4 * 6 = 1.5 Accuracy = 1 - (MSE / Max Squared Error) = 1 - (1.5 / 4) = 0.625

So, our accuracy is 62.5%.

Precision

Another thing to measure is the precision or consistency of the outputs. LLMs are not always consistent. You can test this by running the exact input (prompt, model, model parameters combination) a minimum of 10 times. Ideally, it would produce the same output every time, but due to the nature of LLMs, it may vary. I’ve found that most prompts running against the GPT-4 and Claude 3 models have about an 80–90% precision.

Now that you know how to measure the accuracy and precision of your prompts, you might find that it’s not as accurate as you thought. So, how can you improve it? There are various methods you can try, like prompt engineering, ensembling, and more. I’ll write about the different methods I’ve used to improve accuracy in future posts.

In the meantime, let me know what you think? Are there other ways you’ve used to evaluate your prompts?

Originally published at https://justindra.com.

When I was working on the 12 startups challenge, one of the most common questions I was asked was how I come up with such different ideas. The short answer is that I tend to try and do something completely different from my everyday activities. Let’s dive into one of those experiences.

Last month, I traded my desk and computer for the forest, diving into the world of commercial morel mushroom picking. Hauling 30 to 40 lbs of mushrooms daily through the forest was a radical shift from my usual routine as a software engineer. This experience didn’t just refresh me-it sparked my creativity and provided fertile ground for new ideas.

Here are some lessons I learned from my month in the forests of Northern BC, Canada.

Survival Instincts and Innovation

Living off-grid meant tackling basic survival tasks that most of us take for granted. Hauling drinking water from the closest town (100 km away) or from a nearby river, and chopping firewood to start a fire for a hot shower were daily tasks. Knowing how to handle food to discourage bears and other wildlife from visiting our camp was crucial. These challenges forced me to think on my feet and come up with creative solutions-a skill that’s equally valuable in the startup world.

Physical Work and Mental Clarity

Mushroom picking is physically demanding, involving an estimated 600 to 700 squats per 10 lbs of mushrooms, plus the added challenge of carrying heavy loads across uneven terrain. This physical “treasure hunt” gave my mind the freedom to wander and reflect, often leading to fresh ideas and problem-solving insights. It’s like a meditative break from the mental grind of software development, allowing room for creative thought.

Diverse Encounters, Fresh Perspectives

Meeting people outside my tech bubble-tree planters, rangers, local First Nations members, and fellow pickers-exposed me to different lifestyles and perspectives. These interactions broadened my understanding of various needs and problems, often sparking new ideas that wouldn’t emerge in my usual environment.

If you’ve been curious about how I come up with different startup ideas, the answer is simple: I seek experiences that break my routine. Whether it’s a month in the forest or a new, challenging activity, these experiences inspire creativity by pushing me out of my comfort zone. I highly recommend trying something different-it doesn’t have to be as extreme as living in a forest for a month, but even small changes can lead to significant insights. Here are some other activities I’ve done that are outside the norm:

• Van-life around BC for a few months
• Traveling to different countries
• Volunteering at a music festival
• Woodworking projects around the house
• Go to a networking event for a different industry

So, what’s your next adventure?

For those of you that love numbers, here are some stats from my month in the forest:

• Total mushrooms picked (between 3 of us): approx. 3,000 lbs
• Total distance driven: approx. 3,000 km
• Total days: 29 days
• Most mushrooms picked in a day:
• Me: approx. 55 lbs
• Best of the group: approx. 80 lbs

Originally published at https://justindra.com.

If you’ve been following my 12 startups journey, you might remember that one of my projects is a festival listing website called Festack. One of the issues with such a site is keeping the information up-to-date and I’ve been looking for a way to automate the process of adding new festivals to the site. Web-scraping proofed to be challenging as every single festival has their own custom site and some even change their entire site each year.

Recently, I’ve been exposed more and more to the capabilities of OpenAI ‘s GPT and I thought it might be a good fit for this problem. So I wondered if I could combine some scraping with GPT to get the data I needed. So I went and trialed it out on a few different festival websites, but unfortunately it did not work out. Issues with the data being inconsistent, information being spread across many pages and just the fact that the HTML is quite large and complex.

After talking to a friend of mine, discussing similar issues that she was running into with her project, it came to me. Almost every single music festival has a poster that has most of the information I’d want. So I thought, why not use OCR (Optical Character Recognition) to extract the text from the poster and then use GPT to summarize that information into a structured format that I can use! So, here is the result of that experiment.

Pretty cool huh? I’m pretty happy with the results. I definitely won’t be saving that data directly into the database, as I still find some discrepancies every now and then. But it does get 80% off the work done faster. Alright, let’s get technical now and dive in to how I did it.

Everything here was done in the AWS ecosystem, using SST as the Infrastructure as Code framework.

Firstly, I created a lambda function that would be triggered whenever a new image was uploaded into S3. Using the event details, I then send that to AWS Rekognition to extract the text from the image. Now the text is returned in a JSON format, and it is very verbose, it includes a lot of information like where the text was located in the image, how confident it is that it is the correct text, etc. Through testing, I’ve found that all I needed is just the text itself for each line. So I wrote a small script to clean up the data before sending it.

// Infrastructure code
export const ScraperStack = ({ stack, app }: StackContext) => {
  const db = use(DatabaseStack);

  const openAI = Secret.create(stack, 'OPENAI_API_KEY');

  new Bucket(stack, 'ScraperBucket', {
    notifications: {
      objectCreated: {
        function: {
          handler: 'services/functions/scraper/object-created.handler',
          bind: [db, openAI.OPENAI_API_KEY],
        },
        events: ['object_created'],
        filters: [{ prefix: 'new/' }],
      },
    },
  });
};

// Lambda code to handle when the object is created
// services/functions/scraper/object-created.ts
export const handler: S3Handler = async (evt) => {
  // There might be a few records at a time, so loop through it.
  for (const record of evt.Records) {
    const { key, versionId } = record.s3.object;

    // Detect text
    const ocrResult = await detectText({
      Bucket: record.s3.bucket.name,
      Name: key,
      Version: versionId,
    });

    // Clean out the data
    const cleanedOcrResult = (ocrResult || [])
      .filter((val) => val.Type === 'LINE')
      .map((val) => val.DetectedText);

    // Send to OpenAI
    const res = await parseOCRThroughOpenAI(JSON.stringify(cleanedOcrResult));

    // Do something with the result
    console.log(res);
  }
};

// OCR Function
import {
  DetectTextCommand,
  RekognitionClient,
  S3Object,
} from '@aws-sdk/client-rekognition';

const client = new RekognitionClient({});

export async function detectText(s3Object: S3Object) {
  const command = new DetectTextCommand({
    Image: { S3Object: s3Object },
  });

  const result = await client.send(command);
  return result.TextDetections;
}

Armed with the cleaned up text, I then utilized LangChain to send that text over to OpenAI and summarize it to get some data that I want out of it.

// LangChain Function
export async function parseOCRThroughOpenAI(ocrResult: string) {
  const model = new OpenAI({
    openAIApiKey: Config.OPENAI_API_KEY,
    temperature: 0,
    modelName: 'gpt-3.5-turbo',
  });

  const parser = StructuredOutputParser.fromNamesAndDescriptions({
    name: 'The name of the festival',
    startDate: 'Start date of the festival in ISO-8601 format',
    endDate: 'End date of the festival in ISO-8601 format',
    location: 'The location of the festival',
    website: 'The website of the festival',
    artists: 'An array of artists at the festival',
  });

  const formatInstructions = parser.getFormatInstructions();

  const prompt = new PromptTemplate({
    template:
      'You are a tool that translates OCR results into structured data.\n{formatInstructions}\nOCR Result:\n"""{ocrResult}"""',
    inputVariables: ['ocrResult'],
    partialVariables: { formatInstructions: formatInstructions },
  });

  const input = await prompt.format({
    ocrResult,
  });

  const response = await model.call(input);
  return parser.parse(response);
}

And that’s it! Surprisingly very easy to do! You can now do whatever you want with this data. Currently I’m saving it into the database so that I can review the information and then approve each one, just to monitor the results first.

Let me know what you think about this experiment and if you have any ideas on how I can improve it. Do you have projects that you’d like to test this out on? Reach out to me via LinkedIn or email if you’d like my help.

Originally published at https://justindra.com.

After a few months of having my 12 startups challenge on pause, I’m back at it! Ruru is an AI chat-bot that answers all of your pet-related questions.

Before I get into the details, I just wanted to mention that out of the other 5 ideas I’ve done, this has been by far the fastest one that I’ve gone from idea to launch. It took me about 3 days to build and launch this. So I think all those reps I’ve been doing (and a library of code snippets I’ve accumulated) are starting to pay off! Alright, let’s get into it!

The idea

When I spoke to friends and family that owned pets, they mention that because of the pet-boom from the pandemic, it has been significantly harder to get an appointment with a vet. This makes sense, there is a limited supply of vets and a lot more pets! Since it takes a while for someone to become a vet, understandably, this is going to be an issue for the foreseeable future.

As anyone involved with technology has seen recently, ChatGPT has taken the world by storm. Not just because of the powerful LLM (Large Language Model) behind it, but also due to the simplicity of the interface. It finally delivered what all the chat-bots from a few years ago promised to be. So like many of us, I started looking into it, playing with it, testing out different prompts and after playing with the OpenAI API, I thought this would be a great solution for the problem!

I think Ruru can help people answer the less urgent questions they have about their pets. Whilst making sure that their local vets can handle the more physical nature of their pets issues. I think this is a win-win for everyone involved.

Trivia: Ruru (or morepork) is an owl that is native to New Zealand and Australia. Also, it is the name that I used for a last-mile delivery service that my mate and I tried to push at the start of the pandemic. Since that didn’t pan out, I thought it’d be cool to bring it back for this project.

Competitors and market

The pet-industry has blown up due to the pandemic. And although it’s growth has slowed down since, it is still a growing industry. So, I think that there is a lot of room for Ruru to grow in the future.

In terms of competitors, similar to the human healthcare industry, there are tonne of companies providing a “Chat with a Vet” online service. Most of these will connect you to a vet that you can either chat online, speak on the phone or over a video call. Some provide 24/7 service and others provide it after-hours when your local vet would not be open. I have not found any that provided this service through an AI yet, but I’m on the lookout for one.

I don’t think the technology is there yet to provide a full-service chat-bot that can diagnose your pet’s issues. However, I think we’ve gotta start somewhere!

What I built

I built the Ruru chatbot on top of OpenAI ‘s API. I did have to setup some prompts so that the chatbot would stay in the pet-related questions and to make sure that it does not try to answer questions that are outside of its scope. Especially since one of the bigger issues with ChatGPT has been that it can be quite confident in providing a non-truthful answer.

I also built a simple credit system that measures the number of questions that the user sends through and charges them based on this.

For those interested, here is the tech stack:

• Serverless backend using SST
• Landing page and app is a Vite powered React app
• Payments are handled by Stripe

How will this make money?

As mentioned earlier, I build a credit system, so at the moment the price is $1 per 10 questions. I’m not actually sure if this covers my costs of just running it, but I thought it is a pretty good price to get people interested in it. I decided to go with the usage-based pricing, as I don’t think people will use it consistently for X number of times in a month, but rather they will use it when they need it.

To entice new users, I’ve also set it up to provide the first 10 questions for free. Again, I’m not sure if this is a good idea, but I think it will help people get a feel for it and see if it is something they would use.

In saying all this, I’ll monitor the costs and adjust my pricing accordingly. In the future, I could potentially offer a monthly or annual subscription for unlimited usage. But I’ll see how people use it and see if that’s necessary or not.

How have I marketed this?

So far I’ve just posted this on different platforms:

• Posted on LinkedIn, Facebook and Instagram through my personal account
• Posted on a couple Slack and Discord communities
• Messaged it to some close friends and family
• Posted it up on Futurepedia, an AI tool directory site as well as a few other directories as well

Results

It has been a week since launch, here are the results:

• 500 unique visitors to the site
• 38 users signed up
• 0 paid users
• Received a few messages from people mentioning they liked the idea, provided feedback on bugs and asked questions about how it works

So for the first week, I think this is a good start. I will need to figure out how I might be able to convert people from the free tier to the paid tier.

What’s next?

Alright, what’s next with this. Well, I think this has been a lot fun to work on. I learned a tonne on the AI side of things and starting to grasp that little bit more on how all of this works. So eventually I’d like to move away from using OpenAI’s API and host my own model. Mainly because I want to be able to get it to answer the questions based on data that I know is correct. So will have to find a database of documents around pet care that has been vetted by a vet. If anyone knows where to find this, please let me know!

Other features I think might be useful is providing more context with the AI on the pet. So for example, if you have a dog, it might be useful to know the breed, age, weight, etc. This will help the AI to provide more accurate answers. I think this is something that I’ll have to look into more.

Well, that is it for now. It’s good to be back into building things again. I hope you enjoyed this post and as always, if you have questions, feedback or need some encouragement to launch your own project, please reach out!

On a side note, I’m looking for contract work to help fund these projects until I can get them to a point where they can sustain themselves. You can find me on LinkedIn or email me at hello@justindra.com.

Originally published at https://justindra.com.

If you’ve been following my journey on the 12 startups, you know that I have paused that for a couple months. I’m now about to continue on that journey. But since I haven’t done much coding, I’m a little rusty, so I decided to see if I can build a simple project in 1-day.

The project

I’m a fan of the My First Million podcast, with their host Sam Parr and Shaan Puri. In recent episodes, they started doing their own version of the “Like and Subscribe” call-to-action for their YouTube channel. They call it, “The Gentlemen’s Agreement”.

The agreement is that, they will continue to create content for us the fans, but the fans need to do our part and even it out by clicking that subscribe button. Now, they hammer on that this is a two-sided agreement and that us fans really need to bring our side of the table.

When they first mentioned it, it was pretty funny. But then after doing my part and hitting that subscribe button, it continuously gets mentioned each time, it’s starting to get old. Bringing back the “Like and Subscribe” vibe from other channels. So, this got me thinking, how much content have they created vs the amount of effort we’ve spent as fans in hitting that subscribe button and can we track it?

From there, it comes this project, a simple little website that tracks the total duration of all full episodes of the podcast as well as the total number of subscribers. Based on this, it should be able to determine who is the slacker that needs to pick up their game.

Check out the result at mfm.justindra.com and continue reading to know about how I turned this idea to launch in 1 day.

The process

Requirements

Before I start with any project, I generally start with a list of features. I will just start brainstorming anything that I can think of that would be necessary and then divide that into “must-have” and “nice-to-have” features. So here was my list:

• Must-have
  * Show a description of the agreement
  * Show the data in a clear concise manner
   — Number of Subscribers
   — Total Duration of all published episodes
  * Calculate and show the “Slacker”, whether it’s the host or the fans that need to pick up their slack
  * Update the data daily
• Nice-to-haves
  * Show a graph on the progress since inception and have a cross-over of when we think both-sides will honour their agreement, e.g. a look into the future
  * Let people subscribe and get a weekly email on the latest progress
  * Tweet the current progress each time the data is updated
  * Update the data whenever a new episode is published rather than daily

With these requirements, since I wanted to build this within a day, I also think of whether or not I am able to build those must-haves in 1 day. If I can’t then I will really re-think it and see if there are any other features that I could move to the nice-to-have pile. So I re-iterated over this process until I got a nice set of features that I believe is possible to build in the timeframe I’d like.

Can we get the data we need?

So, with those requirements done, let’s actually see if I can get the data we need. So I spent about 15 minutes reading through YouTube’s documentation to see if it’s at all possible. Looks like it all was, the only issue was getting the duration of the episodes, as they also publish shorter videos that are just clips from their full-episodes. The MFM YouTube channel also hosts a tonne of older videos from HustleCon and other projects that Sam and Shaan worked on in the past. Luckily, I saw that they made a playlist for all the MFM full-episodes, so that was a freebie!

Deciding on a tech stack

This is a pretty simple project, so all I would need is:

• A webpage that is updated each day, should be pretty simple and static
• A CRON job (scheduled function) that runs each day to get the latest data
• A database (if we decide to save the data gathered)

So, based on all this and also my experience in using different tools, I decided to just use my go-to tech-stacks.

• SST — A framework that makes it super easy and fast to build on top of AWS’s serverless offerings
• Astro — A frontend framework that makes really fast websites, especially static sites
• TailwindCSS — A CSS library to help me with styling the site
• AWS DynamoDB — A serverless database by AWS, it’s charged based on usage and requires no effort to setup

If I were to build this with no-code tools, I would probably setup something like Zapier to get the data and then save that into Airtable and use one of the myriad of website builders to build and host the website.

UI Design

I generally like to start of with what the whole thing is going to look like to the user as that may change some of the backend requirements. So I started out with a sketch in my notepad for some ideas on what it may look like. I also took this opportunity to try out Midjourney (an image generation AI) and it’s web design capabilities. You can see some of those iterations below.

In the end, I didn’t end up liking any of the designs for the page, but it did give me some inspiration on what to build. I also asked Midjourney to make me a logo, and it came up with something that I thought was pretty cool. So I chose one, put it into Photoshop to do some adjustments, and we have a logo and a colour scheme.

One other part of the UI design that I always end up spending tonnes of time on is choosing a nice font. I can spend hours just scrolling through a bunch of fonts. So given the time-constraint, I asked ChatGPT for help. This worked out really great! Time-saved!

I then put all that into Figma to give me a rough idea of what the site would look like. I thought it looks pretty good.

Developing the app

Armed with the design, documentation on YouTube’s API, a tech-stack I’d like to use, and a set of requirements that will keep me in check, I got to coding! Without going down into the nitty gritty, here’s at a high-level list of steps that I did.

• Setup SST and Astro boilerplates
• Get API keys for Google and enable the YouTube Data API
• Code and tested the function to get the data
• Build out the webpage by utilizing hard-coded fake data
• Updated the function to save the data in the database
• Connected the webpage to actual data
• Added analytics
• Deploy the app
• Double check to see if I’ve missed anything.

I’ve open-sourced the code on GitHub so you can check it out there if you’re interested.

Conclusion

Once that’s been deployed, I then posted on LinkedIn, Twitter and a Discord group for SST. Still haven’t heard from the hosts about what they thought of this, so we’ll see how it goes!

This was a really fun little project to work on. Let me know what you think! What are some little projects that you’ve launched or would like to launch? I’d love to hear about it!

Originally published at https://justindra.com.

End of November 2022, I saw a pretty cool demo from this AI startup out of San Francisco, and as I was looking through their website, they were hiring. I do not have any AI experience at all other than playing with ChatGPT, but I figured a frontend role likely won’t require any AI experience and I could learn about AI by being surrounded by the experts. This is the first time I’ve ever applied to a job out of San Francisco and although I did not end up with an offer, I thought the experience was quite interesting and worth sharing.

Before I talk about the process, here is a little bit of background context:

• The role — Full-time Frontend Engineer, located in San Francisco, in-office
• The company — A well-funded AI startup, ex-Google founders, “let’s change the world” kinda company
• Interview Timeline — The whole process (including initial application) took approximately 3 months

So, I thought it would be useful to go through this as per the steps I went through from start to finish. I’ve also added whether that step was performed remotely or on-site and if remotely, whether or not it was done asynchronously or synchronously (e.g. video chat). Here are the full-steps if you’d like to jump ahead.

1. Application (Remote Async)
2. Screening Call (Remote Sync)
3. Take Home Assessment (Remote Async)
4. Technical Screen — Frontend (Remote Sync)
5. Technical Screen — Algorithms and Coding (On-site) x2
6. Systems Design (On-site)
7. Culture Fit (On-site)
8. Technical Screen — Frontend (Remote Sync)
9. Feedback Session (Remote Sync)
10. Final Thoughts

1. Application (Remote Async)

The application was a pretty standard form, with submit your resume, LinkedIn, portfolio, as well as a few general screening questions, e.g. Why do you want to work for us etc. So I submitted mine and waited. No confirmation of receipt, and no communication until approx. 2 weeks, when I received an email from their recruiter mentioning to book a session with their hiring manager.

2. Screening Call (Remote Sync)

The screening call was with the person who would be my manager if I was offered the role. I thought that this was really great as it allowed me to ask more specific questions around the role and what his expectations are around work-life balance and general team structure. I was also able to gauge whether or not I would be comfortable working with this person. I’ve had other screening calls before with an internal recruiter and it has generally not been a great experience as they aren’t generally able to answer specific questions about the team and what you would actually be doing in the role.

3. Take Home Assessment (Remote Async)

For the next step, I was given a choice whether to complete a HackerRank challenge or a take-home assessment building a particular product. I chose the take-home assessment as I don’t think the HackerRank challenge really showcases your skills or more importantly, how you would actually build and deploy a project.

The task provided was basically using OpenAI ‘s API to power a basic calendar app in any framework of my choosing. I thought they did really well in providing the scope as well as the basic repository structure for me to use. They also provided a set of functions that interacted with OpenAI’s API, so it was focused on the frontend itself. Submission itself was just a zipped folder of the repo that I emailed back to them, nice and easy.

4. Technical Screen — Frontend (Remote Sync)

The next step, was a Frontend coding interview. I thought this was an odd thing to have after the take home assessment. Especially as it was not in any way related to the take-home assessment.

I personally thought that this step was ill-prepared. The assessment, which was to create a React component that interacted with OpenAI’s API felt repetitive. But, compared to the take-home assessment, the library provided was a little more open-ended and some AI prompt design was required. For anyone who has played with ChatGPT, probably knows adjusting prompts takes some time and is an art in itself. Not something that can easily be done in a 45-min slot as part of coding other things.

The other issue was the fact that it seems like the interviewer seems quite early in their interviewing journey. For someone who has been on both sides, I think it’s a great opportunity given to him by the company to learn how to interview others. But ideally, they should have really had another person there to guide him through the interview process or had him assess a more prepared assessment.

5. Technical Screen — Algorithms and Coding (On-site) x2

The next few steps were done in a single day in their San Francisco office. The day started off with two algorithms and coding problems with 2 different interviewers, other members of the team. The first was performed on a shared IDE and the second on a combination of a whiteboard and my own IDE.

Generally speaking, I’m not a fan of these types of interviews as it doesn’t translate well into your day-to-day. But it does show how well you will work with different people in the team. Something I noticed here was that neither of the interviewers are frontend developers and so these problems were not specifically frontend problems, just general coding challenges. I did think that they did well in designing it to mimic some of the issues they may face in the Large Language Model (LLM) space that powers most AI tools like ChatGPT.

6. Systems Design (On-site)

The next on-site step is a systems design interview, where a potential feature was presented and the interviewer is acting as the product owner, customer and all other stakeholders. The interview was with the person who will be my manager if the role was offered.

I thought that this was a good interview to test out the way I thought of problems and potential ways to solve it. In saying that, I am someone who tend to think of things on my own time while having a shower, going on a hike or generally just sitting there with my cup of tea before showing a solution so it was a little nerve-wracking to do this train of thought in front of someone. I would be curious if there were more inclusive ways to perform this type of interviews whilst still getting the benefit of seeing the interviewees process.

As a side note, at this point I was a little surprised that there has been no mention of what I had built for the take-home assessment. No questions, feedback or even any acknowledgment of it.

7. Culture Fit (On-site)

The last on-site step was a culture fit assessment. At least that’s what they called it. I personally did not think it was based on my experience as an interviewer and interviewee of other culture-fit interviews I’ve had.

The interviewer was someone that is part of Business Development and Partnerships, a change from the other interviewers who were all from the development team. The questions that came were the standard, rigid questions around where I see myself in X years, what I’d like to get out of this role, or what feedback have I received in the past, etc.

My expectations for culture-fit is whether or not I shared the same values as most of the people at the company. For some companies, they have their values written down, for others, it is an unspoken set of values. This is why I think that the best culture-fit interview to do is to have a round-table style interview. Place a few people from the company to sit in a table with the interviewee and just have a chat around any topic. You can generally tell pretty quickly if this person fits the mould you’d like to shape or not. What are your thoughts around this? Does culture-fit mean a different thing in SF than elsewhere?

8. Technical Screen — Frontend (Remote Sync)

The next day, I received a phone call to see if I’d be ok to take another interview with someone they had just hired as their first frontend engineer. This explains many of the downsides of some of the interviews I’ve had. I agreed as I’d like to meet the person that I will likely be spending my time with the most if I was offered the role. Plus, it would be interesting to speak to someone with a bit more frontend knowledge.

The first part of this step was again another coding session. This time we are creating a recursive React component, an interesting challenge but nothing too complicated.

The rest of the interview turned into a conversation around the challenges that they face with the chrome extension and how it might be solved. Although this just happened naturally, I think this was a really great example of taking an actual frontend problem and discussing potential solutions that could address it. Which is what most engineers do. The coding itself is secondary to these sort of problem solving.

Although the interview went almost an hour overtime, I thought it was great and is what I would have expected the other technical screens to consist of.

9. Feedback Session (Remote Sync)

Ultimately, I did not receive the offer. I’m a little disappointed but just like anything, I try to use it as a learning experience. So like improving any product, I asked for feedback.

They agreed to a feedback session over zoom with their internal recruiter. Unfortunately, the feedback was disappointing. It was a very vague feedback that doesn’t really allow me to set up some actionable steps to take. And because the feedback session was with their internal recruiter and not any of the 6 people that performed the interviews, I wasn’t able to clarify or expand on any of this. This is probably more disappointing than the results itself, as how do you improve without feedback?

Final Thoughts

My final thoughts on this whole experience is that, it has been very interesting. I clearly need to improve somethings to be worthy of this particular role. Just need to find out what that is. Also, unsure if this many steps for a frontend engineer is normal for companies in San Francisco. It’s definitely double to what I am used to. Let me know your thoughts on this and maybe share your experience on your interview journey!

Originally published at https://justindra.com.

This month I decided to hit pause on my 12 startups challenge journey. I spent the last few weeks making this decision and I’d like to talk about why I decided to pause.

A quick recap on the projects:

• #1 Beaco — A physical presence tracker for Canadian Permanent Residents
• #2 Festack — A list of different music festivals
• #3 ifoundthis — Get your lost items back!
• #4 egami — Generate your social share images with no effort
• #5 OneEPK — Helping artists create, manage and share their EPKs

Original goals

When I started this, I wrote down a few goals that I wanted to achieve. As part of my decision making process, I reviewed these goals and how I have progressed against them.

To launch fast!

The main reason I started this, is because I realized that I had many ideas that never saw the light of day. So I wanted to launch things, and do them quickly. Fitting it into a timeline that forces me to forgo the perfect product and actually validating it. Now that I’ve launched multiple things, I have found a process that I follow that works for me to quickly research, validate and launch an idea. This process came from what I knew, what I’ve learned from others and what I’ve learned as I test them out over these ideas.

On the tech-stack side, I’ve also gotten to try some new tech-stacks, especially the no-code tools, and have a bit of a general set of frameworks/libraries that I will likely use each time I launch something.

Find that passion

Every time I look into starting a startup or business, everyone continuously mentions 2 things, work on something you are passionate about and focus on that. I am someone who was always interested in too many things at the same time. I studied Mechatronics Engineering because I couldn’t decide whether I wanted to do mechanical, electronics or software, luckily they started a degree to combine them all. So I wanted to launch a few things in different areas that I am interested in and see which I find to be more exciting than the others.

After launching #4, I decided to work more on the music industry. Because I love going to festivals, seeing live music, everytime I code (and writing this article), I’m constantly listening to music. So I’d like to contribute something towards that industry and especially to the artists that make these amazing pieces.

Get better at marketing

When I started, I’ve had some experience marketing things for the previous startups I’ve worked for. I didn’t always agree that we took the right approaches, but had no way of knowing if what I thought was right would have worked or not. This time around, I got to test my theories, and boy did I learn a lot. I learned why certain things have worked and what other things haven’t.

Am I now an expert? Definitely not! I’ve only touched on so little, and having had to do it with multiple projects that require different testing, I don’t think I’ve fully comprehended all the learnings.

Stretch Goal: Generate MRR

As a stretch goal, I wanted to achieve ramen-profitability in monthly recurring revenue. I have not gotten there. I made a total of USD$45 and some IOU beers, and I’ve spent a lot more in just domain names.

Making the decisions

I achieved my main goal, which was to launch things quickly and get over my perfectionist-side. I also found something that I am more interested in than others. But when it comes to marketing and generating MRR, not so much. As I look to different projects that people have built and reading about their journey, I can tell that nothing good takes a month to build. Of course there are exceptions to this, but I’m not going to fantasize to be the exception.

On another note, I am currently applying to a few jobs and I have been travelling between Canada and the USA since December. Continuing onto Australia and New Zealand in February. Although the travel has been great in getting inspirations and meeting amazing people to talk to about these projects, I’ve found that I am struggling to strike a healthy balance between enjoying my travel and keeping onto these projects.

Based on the progress on my goals, what others have done and my personal situation, I decided that I should concentrate on a single project for the next three months. In three months time, I will re-evaluate what has happened and whether or not I will hit play again on this challenge.

What’s next?

So, for the next three months, I will be concentrating on OneEPK. I chose OneEPK as it is more aligned with what I am most interested in and I have received the most feedback compared to others. So although no paying users yet, it is something that struck a chord with enough people that I’d like to continue building on it.

For those of you that have been following and are wanting more, these are others I’ve met that are currently doing this challenge. I’d recommend you follow their journey.

• Meelis Ojasild
• Akshay Mattoo

Originally published at https://justindra.com.

OneEPK is the fifth startup in my 12 startups challenge. It is a platform for DJs, performers, musicians and artists to create and manage their online EPK with ease.

The idea

As an artist, if you wanted to play at larger venues, music festivals or be considered by labels, one of the items they ask of you is an EPK. An EPK, or electronic press kit, is a collection of promotional materials and information about an artist or band that is used to promote their music and upcoming events or releases. It is typically a digital document, often in the form of a website or PDF, that includes things like a bio, photos, videos, music samples, press clippings, and contact information. In other words, it is an artist’ version of a résumé.

For most emerging artist, creating and then maintaining their EPK is one more thing that takes them away from what they should be working on, their music. So I wanted to create a platform that makes it a bit easier for them to do so.

Competitors and market

To see if this is something that is worth building, I spoke to a few DJs that I know of as well as created a survey that I posted on reddit and shared through my own socials. I also had one amazing person reach out to me via LinkedIn who provided a really great insight into the music industry and how something like this could be a great stepping stone into larger things.

There are a couple competitors out in the market, and these are the issues that I’ve found both from testing it out as well as from my conversations.

• It was created a while back and has not been updated
• Not mobile-friendly
• There’s a focus on expecting everyone to be on their platform, therefore they shy away from integrations to other platforms

Based on these, I believe it should be possible to create a better version of what is already out there and attract some customers.

What I built

Creating a platform that does everything an EPK can possibly consist of is not possible for me in a month’s time with other commitments. So I reduced down that feature set to something more manageable but it should still provide some value right of the bat.

I made a platform where an artist can sign in using either Google or Spotify and then create an artist’ page. As part of the creation process, they can choose a unique link that they can use to share anywhere. This page can include their stage name, bio, a cover photo, contact details, links to other social channels and an embedded Soundcloud/Spotify playlist.

If you’d like to see a demo, there is one on the OneEPK homepage.

For those interested, here is the tech stack:

• Serverless backend using SST
• Landing page and public site is a SSR (Server-Side Rendered) Astro site
• Web app is a Vite powered React app

How will this make money?

This will not make money in the near future. I made a decision that I wanted to create a tool that will support artists and so I did not want to charge them to use this tool. I am hoping that my other projects that I have and will launch will be able to support this project to keep it free for all artists. If I had to charge artists, it will likely be to do with merchandising where I can take a percentage of any income created through the platform, or potentially by selling access or data to the record labels that manage these artists.

How have I marketed this?

So far here is how I’ve launched this:

• Posted on LinkedIn, Reddit and my own socials
• Sent invites to DJ friends to see if they would try it
• I went and created a basic EPK for different DJs that I found on instagram and then emailed them a link to see if they would be interested in using the platform themselves

Results

After 1 week since launch, here are the results:

• 4 users signed up
• Received feedback that I will need to allow for extra details, primarily the statistics from different social platforms in one place

What’s next?

There are certain features that I may need to add in, the primary one is statistics from other social platforms, so that they can see how many followers they have on instagram, twitter, spotify, etc. The next one is events, being able to let artist add in their previous and upcoming events. So that their fans can see what is coming up soon. So I will likely be adding these in at some point. The other thing would be to integrate this better with Festack, my earlier project, so that it can better share that SEO love.

That’s number 5 out of the oven! I’m almost half-way and it’s been amazing! I’m really enjoying this challenge and hopefully you are too! If you’d like to chat about your own project, or ask any questions around my processes, please do reach out!

Originally published at https://justindra.com.

I have now spent just under 5 months working on the 12 Startups Challenge. I have launched 4 different ideas and although a little delayed, will be working on my 5th soon. A few people have asked me about what my process is on launching each idea. So I thought I’d write it down and hopefully it can help others in launching their ideas.

Here is a quick summary of the steps, and I’ll dive a little deeper down below.

1. Find an Idea
2. Competitor and Market Analysis
3. Idea Validation
4. Plan Out Your MVP
5. Build That MVP
6. Launch It!
7. Review and Iterate

As you go through these steps, remember that you can spend as much or as little time as you’d like. Depending on the idea, you could spend an hour on each and launch within a day!

1. Find an Idea

To get started, you of course need an idea or a problem to solve. So far I’ve just been working on problems that I’ve personally encountered. Although, since I started the challenge and posting about my progress, I’ve also had several people come to me with their ideas too. If you don’t have an idea to work on, there are a tonne of ways you can come up with it and Google is your friend here. You can even Google “Business Ideas” and you’ll probably get a tonne of ideas on other people’s list. Or you can just take one of my odd-ideas that I post on LinkedIn each Monday.

2. Competitor and Market Analysis (optional)

This is an optional step, mostly because it depends on the idea and depends on how much you care about the success of the project vs just practicing launching an idea. I generally do a quick Googling and research on different sub-reddits or facebook groups just to get a gauge on how big the market is and if there are any other competitors in the space. Having competitors does not mean you shouldn’t work on it, if anything it means that there is a need for that idea. Competitors can show you what features are important to people or based on reviews of the competitor you can also see if they have missed some features that people are asking for.

3. Idea Validation (optional)

Ideally you’d want to validate the idea you are working on to see if at least one person would use it (and hopefully pay for it). The nice thing about working on your own problems is that you have at least 1 user, which is yourself. On this step, I would reach out to people who I think would benefit from my idea. I would ask them some questions around how they are currently dealing with the problem you are wanting to solve. I have found The Mom Test to be a very useful resource on asking the right questions to validate your idea.

4. Plan Out Your MVP

This is it, you’ve got an idea, you’ve spoken to people who are interested in it, now you need to work out what should be included in your MVP (Minimum Viable Product). Here is my general process on how I approach this.

1. Start writing down your features, requirements or user stories as granular as possible: e.g. User needs to be able to sign in, admin needs to be able to see a users profile
2. For each requirement, indicate whether this is a must-have, or a nice-to-have. Really knuckle down on this! As an example, does your user really need to be able to reset their password on Day 1? Probably not, you can probably just manually reset it for them if they forgot their password.
3. Concentrate on the must-haves from above, and re-do that prioritization, but this time, look at it with a time-constraint. If I only have 1 week to build this, what can I build out of that list that actually solves the problem, after that reduce it down to 1/2 a week, and then to 1 day and so on.
4. By now you should have a pretty good idea of which features will solve the problem and which are just nice-to-haves. You also now have a pseudo roadmap of what to build each day. Ideally after that 1st day, you can reach back out to your potential customers and get them to start using it.

5. Build That MVP

Pretty self-explanatory here, but go build it! Remember to concentrate on the really important features that you worked out in the previous step. This is also where you can get really creative. Not everything has to be built with code. Uber’s MVP was just a phone number you can text to get picked up, YNAB’s MVP was an excel spreadsheet that you can download. There are also so many no-code tools out there today that can help you spin things up really fast. Basically get creative and use the skills you have to launch things as fast as possible.

6. Launch It!

This is it, you’ve got your shiny new MVP. Launch it, deploy it, share it! Go back to the initial people that you spoke to and send it to them and ask them for feedback. Regardless of the result, just remember that you launched something! That’s a great thing, so celebrate that win!

7. Review and Iterate

Now that you’ve launched your idea, review your results. What feedback did you get from your initial users? Remember that no feedback is also a form of feedback. Work out what went well, what went wrong, and then iterate over. You now have a choice to continue with this idea, keep talking to people to find out what else can be done, or you can call it and move onto the next idea. Either way, you should have learned some new skills, talked to some interesting people and you turned your idea into reality! Now, that’s super cool!

This is the process I have followed and adjusted over the last 4 ideas I launched, but it may change overtime as I learn new things. I do hope that this breakdown can show you how easy it is to get started and that you can go and launch your ideas pretty quickly! If you have any questions or feedback, feel free to reach out to me.

Originally published at https://justindra.com.

egami.io is the 4th startup in my 12 startups challenge. It allows content creators to save time on generating social media images or og:image for their blogs.

The idea

An og:image or a social media share image is an image that is used by many social media providers to generate a preview whenever you post a link to your site. There's a great explanation of it on freecodecamp that I recommend you to check out.

One of my earlier projects, festack, is a festival listing site which would benefit from having a nice looking og:image. I originally started working on creating a template in Figma so I can create them, but with more than 200 festivals already listed there, this is turning out to be a bit of a chore. So I thought this would be a great little product, to be able to generate these images based on a template.

Trivia — What does egami have to do with social media images? What does it even mean?

I’m Australian, so I thought it would be quite funny to request an image by saying “oi! image!”. Unfortunately, .image domains don't exist, and image.com is not available. So I reversed it and now we have

It was an available domain name and I thought it was pretty cool.😛

Competitors and other options

There is a pretty big potential for this product as there are approximately 572 million blogs out there in the world and 7 million blog posts are published each day. Outside of blogs, there are also listing sites, such as job listing sites, real estate and many others that may benefit from this.

I also found a few competitors in this space, with some different takes on this exact issue. So why do I think I could still get customers? Well, the market size is so large and most of the available competitors combine their offering with other social media tools which means that their pricing is a bit unaffordable if you are a smaller blogger. So I intend to target the smaller size businesses and bloggers that generate a little less content.

What I built

So far, I’ve built a system where you can simply add the following and adjust the project-name, template-name and add a few parameters to adjust it.

<meta
  property="og:image"
  content="https://og.egami.io/[project-name]/[template-name].png?[query parameters]" />

Here’s an example that I am using for this blog post:

<meta
  property="og:image"
  content="https://og.egami.io/justindra/post.png?title=Startup%20%234%20-%20egami.io&description=A%20debrief%20on%20egami.io%2C%20a%20social%20media%20sharing%20image%20generator" />

And here is the result:

So how did I build the MVP? I used the following tech stack:

• Image Generation — I used Puppeteer to run Chrome on AWS Lambda and generate the image based on the provided request. Which is then saved in an S3 bucket and also cached on a CDN for a year. To insure that other than the initial generation, it will be returned at lightning speed.
• Landing Page — Like this blog and my other products, I’m really enjoying using Astro to generate static sites, so I’ve used it here to generate the landing page for this product too!
• Lastly, I used SST to deploy all of this on AWS.

How will this make money?

I’ve decided on a $10 per month pricing or $100 per year for the generation of these images. I chose this price based on the fact that this would be the price I would be willing to pay myself. As I gather more feedback from users, and once I get enough traffic, I may need to adjust this pricing to account for different features or infrastructure costs. But ideally I’d like to keep it at this price point.

How have I marketed this?

I am currently marketing this via a few channels:

• Community Sharing — I’ve posted this on a few communities that have a few personal bloggers to see if anyone would be interested in signing up
• Cold emails — I’ve cold emailed a few of the bloggers that I personally follow as well as some job listing sites that I’ve used in the past
• Sharing — Currently some templates for beta users has the “Generated by egami.io” mark on it, I may end up opening up a free-tier to allow those that don’t have enough volume to use this and provide some free promotion for me.

Results

Unfortunately, I have received interest but not enough to get a paying user. The feedback I’ve received is that although this is a problem that people would like solved, it is not necessarily a problem they would pay for.

Other competitors have bundled this together with other offerings in automating image, pdf and report generations. As well as tackling a few more problems in the social media space instead of just the image generation. I may look into this again at some point and see what is exactly involved in building these extra features that may help users convert.

So for the time being, the current users are just myself through all my sites, but the primary user is festack as that was the main reason I wanted to build this.

What’s next?

As mentioned above, other competitors have extra features built on top of the image generation and I may be looking to do this. Primarily around a self-serve process so that potential users can test it out quickly and see if it is something they’d like to use.

That’s the 4th one done and I’m actually quite surprised that I made it this far! I’m really enjoying this challenge and hopefully you are too! If you’d like to chat about your own project, or ask any questions around my processes, please do reach out!

Originally published at https://justindra.com.