In the fast-paced, data-intensive environment of real estate, the management and retrieval of extensive project-related data pose significant challenges. Real estate firms generate a vast array of documents at every stage of their operations — from land feasibility studies and market research to architectural designs and compliance reports. The sheer volume and diversity of this data make traditional storage and retrieval methods inefficient, often leading to delays in decision-making and missed opportunities.

To address these challenges, we have developed a Custom LLM-Powered Knowledge Base using Gemma for text vectorization and MongoDB for storing and querying vector data. This innovative solution allows real estate firms to quickly access, query, and generate insights from their extensive document repositories, transforming how they manage and utilize their data.

Understanding the challenge

Real estate operations involve a multitude of processes, each generating a significant number of documents. Some key document types include:

• Land Feasibility Studies
• Zoning and Land Use Plans
• Market Research Reports
• Project Feasibility Reports
• Environmental Impact Assessments
• Building Plan Approvals

The challenge lies not only in storing these documents but also in efficiently retrieving relevant information when needed. Traditional document management systems, while useful, often fall short in handling unstructured data or performing complex queries across large datasets.

Here we see an opportunity to use LLM’s capability to make a queryable knowledge base using those documents.

Our solution

Our solution leverages LLM’s capabilities to create a powerful knowledge base that can understand, process, and respond to natural language queries. By integrating Gemma (a family of lightweight, state-of-the-art models), with MongoDB Vector Database, we have built a system that efficiently stores, retrieves, and generates insights from vast amounts of data.

Extracting Text from PDF

We use PyMuPDF to convert PDF documents into plain text format. This step is crucial as it transforms unstructured data into a format that can be processed by the model.

import fitz  # PyMuPDF

def extract_text_from_pdf(pdf_path):
    doc = fitz.open(pdf_path)
    text = ""
    for page_num in range(len(doc)):
        page = doc.load_page(page_num)
        text += page.get_text()
    return text

pdf_path = "path/to/your/document.pdf"
document_text = extract_text_from_pdf(pdf_path)

To support vast majority of data types such as pdf, sheets, drawings and etc. We have used different solutions to convert and extract meaningful texts out of those data files.

Vectorizing the Text using Gemma

The extracted text is then vectorized using Gemma. Vectorization involves converting the text into numerical vectors that represent the semantic meaning of the content. These vectors are essential for performing similarity searches and for understanding the context of queries.

import os
from google.cloud import aiplatform

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "<service-json>"

aiplatform.init(
    project="<projectId>",
    location="<region>"
)

def vectorize_text_with_gemma(text):
    gemma_input = {"instances": [{"content": text}]}
    response = model_endpoint.predict(instances=gemma_input)
    embeddings = response.predictions[0]['embedding']
    return embeddings.numpy() if hasattr(embeddings, 'numpy') else embeddings

document_vector = vectorize_text_with_gemma(document_text)

Storing vectors in MongoDB

Once the text is vectorized, we store the vectors in MongoDB Vector Database. MongoDB is ideal for this purpose due to its scalability and support for complex queries. Storing vectors in MongoDB allows for efficient indexing and retrieval, which is crucial for processing large datasets.

from pymongo import MongoClient

client = MongoClient("mongodb://localhost:27017/")
db = client['real_estate_projects']
collection = db['documents']

document = {
    "project_id": "123",
    "type": "budget",
    "content": document_text,
    "vector": document_vector.tolist()
}

collection.insert_one(document)

The above pipeline stores the converted data into Mongo DB in a vector format also known as Data Ingestion Pipeline.

Each time a new dataset is generated, we utilize the Data Ingestion Pipeline to update our Vector Database. This approach ensures that the system remains scalable and efficient, seamlessly supporting ongoing data updates and insertions.

Making a query

Once our Vector Database is configured and populated with all the necessary data, we proceed to build a query interface. This setup ensures that the system is ready to handle complex queries efficiently, leveraging the structured data stored in the database.

We then develop a simple application that allows users to input their queries. The application uses our solution, powered by a Large Language Model (LLM), to process these queries and provide accurate and relevant information in response. This approach streamlines the process of retrieving critical data, making it accessible and actionable for users.

To illustrate this process, we here is a step-by-step example of how data is queried and how the system delivers precise answers based on the input provided.

Query processing and Vectorization

When a user submits a query, it is processed and vectorized using Gemma, similar to how the documents were vectorized. This ensures that the query is represented in the same vector space, allowing for accurate similarity searches.

def vectorize_query_with_gemma(query):
    gemma_input = {"instances": [{"content": query}]}
    response = model_endpoint.predict(instances=gemma_input)
    embeddings = response.predictions[0]['embedding']
    return embeddings.numpy() if hasattr(embeddings, 'numpy') else embeddings

query = "What are the compliance requirements to develop a residential project in Hyderabad?"
query_vector = vectorize_query_with_gemma(query)

Retrieving Relevant Data from MongoDB

The vectorized query is then used to perform a similarity search in MongoDB. This search retrieves the most relevant documents based on their vector similarity to the query.

def retrieve_relevant_documents(query_vector, collection, k=5):
    query_vector_list = query_vector.tolist()
    results = collection.aggregate([
        {
            "$search": {
                "knnBeta": {
                    "vector": query_vector_list,
                    "path": "vector",
                    "k": k
                }
            }
        }
    ])
    return list(results)

retrieved_documents = retrieve_relevant_documents(query_vector, collection)

Generating responses with Gemma

The final step involves generating a coherent response based on the retrieved documents. Gemma uses the context from these documents to generate an answer that is relevant and accurate.

def generate_response_with_gemma(retrieved_documents, query):
    context = " ".join([doc['content'] for doc in retrieved_documents])
    gemma_input = {
        "instances": [{"prompt": f"Context: {context}\nQuery: {query}"}]
    }
    response = model_endpoint.predict(instances=gemma_input)
    generated_text = response.predictions[0]['text']
    return generated_text

response = generate_response_with_gemma(retrieved_documents, query)
print("Generated Response:", response)

This process can be referred to as the query pipeline, where we take the user’s text input, convert it into a vector using Gemma, and query our Vector Database to identify the nodes most relevant to the user’s query.

Looking Ahead

To further enhance the system, we plan to implement several improvements:

1. Data Processing Enhancements: Advanced data cleaning, segmentation, and metadata enrichment to improve the quality of the input data.

2. Vectorization 2.0: Fine-tuning the Gemma model, optimizing embeddings, and batch processing to better capture the nuances of the text.

3. Optimizing Storage: Implementing better indexing strategies, sharding, and data archival to handle growing data volumes more efficiently.

4. Query Enhancements: Incorporating contextual understanding and a feedback loop to refine the accuracy of query responses.

5. Improving Response Generation: Using template-based responses, user personalization, and better error handling to provide more accurate and user-friendly outputs.

The solution presented in this case study is known as Retrieval-Augmented Generation (RAG), where we optimize the process of generating responses using an LLM by sending both the query and relevant context. One limitation is the token size (Query + Context) that can be sent to a specific model to generate responses. RAG addresses this by optimizing context retrieval through an external vector database, making the process more accurate, efficient, and scalable.

Conclusion

Our Custom LLM-Powered Knowledge Base represents a significant advancement in how real estate firms can manage and utilize their data. By integrating Gemma with MongoDB’s advanced vector search capabilities, we have created a system that not only stores vast amounts of data but also makes it accessible and actionable in ways that were previously unattainable. As we continue to refine and expand this solution, we anticipate it will have a profound impact on the efficiency and effectiveness of real estate operations.

For those interested in exploring the technical details or implementing a similar solution, we invite you to visit our GitHub repository or contact us directly. We are also actively hiring — join us in building the future of AI-driven knowledge management!

This detailed article serves as a comprehensive overview of your project, including the problem it solves, the technical approach, and the specific code implementations used. It is designed to resonate with a technical audience who would be interested in the intricacies of your solution and its benefits to the real estate industry.

Case Study — Custom LLM Powered Knowledge Base for Real Estate Industry using Gemma and MongoDB was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here we will see the process of Next.js app deployment using AWS Lambda and S3. It has good scalability and is a cost-saving approach in serverless architecture. Let us have some details about the role of Lambda and S3.

AWS Lambda — 
In this deployment, AWS Lambda is responsible for creating SSR pages dynamically by processing server-side props of the next.js app.

S3 Bucket — 
We will use the S3 bucket to serve all static content of the website like JS, CSS, Media, etc. AWS Lambda has a limit for its size, so we can not put large static files there.

Let's start step by step process to deploy your Next.js app using this process. We will use an AWS Lambda Web Adapter to convert and process the HTTP request received on the lambda function and transfer it to our next.js app.
https://github.com/awslabs/aws-lambda-web-adapter

The basic structure of this process is as below.

1. Build your app in a standalone format.
   Here are some configuration updates for your next.config.js

Here output format is standalone so it will create the standalone app in the build process to deploy anywhere using node. Now you will see a standalone folder inside .next after the build finishes.
Here assetPrefix is used to point all static files to S3 bucket.

Now run next.js build command.

npm run build

It will produce .next folder with the output below —

2. Deploy static contents on S3 Bucket
Copy static foder form .next to your s3 bucket inside ${BUILD_VERSION}/_next/
Now all your static files should be accessible using bucket url like:
https://${s3Bucket}.s3.${awsRegion}.amazonaws.com/${buildVersion}/_next/static/media/51ed15f9841b9f9d-s.woff2

3. Deploy the standalone app to AWS Lambda
Create AWS Lambda function for app deployment.
https://ap-south-1.console.aws.amazon.com/lambda/home?region=ap-south-1#/create/function
Provide the function name and in Advance Setting Check “Enable Function URL”

Add AWS Lambda Web Adapter to our lambda function by using “Add a layer” option.

Select “Specify an ARN” and fill:
arn:aws:lambda:${AWS::Region}:753240598075:layer:LambdaAdapterLayerX86:22
update aws region in the above ARN.

Now AWS Lambda Web Adapter is ready to use. Now update “Runtime settings” handler. Set handler as server.sh.

Before deploying the standalone folder to AWS Lambda we need to create the server.sh file inside the standalone directory and put the execution entry code.

#!/bin/bash
# Execute server.js for SSR pages
exec node server.js

Now copy the public folder from the root of your next.js app into .next/standalone for server public content on SSR.

cp -r public/. .next/standalone/public

Here your standalone app is ready to deploy on AWS lambda, create a zip of contents inside the standalone.

cd .next/standalone
zip -r deploy.zip .

Now upload deploy.zip file in AWS Lambda function.

In the last, we need to set up environment variables that are used in our app. To run the application we need to set up two environment variables initially.

AWS_LAMBDA_EXEC_WRAPPER = /opt/bootstrap
PORT = 8000

Our deployment is finished here, you can test now using the Function URL.

You can see logs in Monitor -> View Cloud Logs to see SSR logs.

The Next.js app is ready to use. To deploy your app just need to set the Function URL to Cloudfront Origin.

Conclusion

Here I was implying the Next.js app deployment that is easy to maintain, watch, and scale. After setting up this infrastructure it is easy to create a CI/CD pipeline using any CI/CD tool.

Deploy Next.js app with SSR using AWS Lambda was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

I have created a class FileUploader that usage Retrofit Library for connecting to API. To upload the file the code skeleton is as follow:

fileUploader.uploadFiles() function takes 5 arguments.

1. API endpoint

2. API file key

3. An array of the File object (Files to be upload)

4. Callbacks

5. Authorization Token (Optional, pass it API require auth token)

A full example of MainActitity.java is as below:

ApiClient.java

FileUploader.java

You can download the full working code sample from GitHub.

sheetalkumar105/retrofit-multiple-file-upload-with-progress

For any other assistance, you can visit DevStudioOnline.

Retrofit multiple file upload with progress in android

Retrofit multiple file upload with progress in android was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

Desktop applications are still a force to reckon with even though there’s been a steady decline in the number of applications created thanks to the growing number of browsers and mobile devices; you can argue that desktop applications are still useful for many reasons.

Desktop applications are ever-present on your computer, connect better thanks to shortcuts and they offer faster notifications for those who delight in lightning speed.

The creation of desktop applications, on the other hand, is not as fanciful as the applications it creates. Using JavaScript, you will be able to create a codebase and package for each operating system with ease. A new method works. Nowadays, Electron JS is the tool used more often along with NW.JS.

What is Electron?

The electron as a framework is used for creating desktop applications in collaboration with other popular technologies such as JavaScript and CSS. It does the dirty work so you can focus on the significant areas of your application.

Electron J.S is open source and works as a cross-platform framework, which means it is compatible with Mac, Windows and Linux. Everything you need to hardcode a desktop application is inbuilt to the Electron framework including updates, notifications, and debugging features.

This framework is used by lots of developers, and this is due in part to some reasons chief of which is the runtime it provides to build desktop applications using pure JavaScript.

Once you fire up this application, a process kickstarts which interacts with the graphical user interface of the operating system. It does this by creating a file called main.js followed by application windows which are filled with rendered web pages with the power to interact with the GUI of the installed operating system.

Electron J.S is more than a framework. It is part of the computer revolution.

Electron Js has been powering some of the most famous desktop applications, and there are no signs of it slowing down anytime soon.

Here are some popular desktop applications that were built on the Electron platform:

Torrent Desktop Apps

When Torrent first broke out in, the whole world broke out into a frenzy. Today, we have millions of people all across the globe using torrent websites to download and share files securely. Todays, torrent files can be downloaded on a mobile phone or to your desktop computers.

There are lots of mobile apps and desktop apps for torrent websites but the most intuitive has to be the WebTorrent app which is built on Electron JS as their basic developing program.

WebTorrent also happens to be the first torrent client that works in the browser and on the desktop. It’s written in JavaScript, and it handles peer-to-peer transport through WebRTC. It works by linking the user to a decentralized browser-to-browser network that ensures the efficient sharing and transfer of files without any plugins or extensions being installed. Thanks to what it’s built on, Electron, the WebTorrent desktop version runs very fast, is lightweight and ad-free ensuring the user enjoys the experience every time it’s’ being used.

CMS Desktop Apps

Content Management Systems have simplified the ease at which digital content is created and managed. CMS systems differ based on the ease at which they can be installed, used, and extended.

A list of the popular content management systems used globally includes Wordpress, Drupal, Joomla, and Alfresco. Wordpress remains the most popular CMS platform, used primarily for blogging thanks to the documentation.

The WordPress Desktop application is built on Electron, and it is built on an open-source framework that makes it easier to manage Wordpress content.

While most users might access WordPress more from their browsers, WordPress for Desktop is an entirely new seamless and cross-platform experience that allows the users to focus more on content creation without all of the distractions that comes with browsers.

WordPress uses Electron due to its minimal load times allowing the writer to get its work done without music hassles. Built on Electron as the framework and JavaScript using React as its primary language.

Ghost is another open-source platform that allows users to build and run their private online publications. It’s not as popular as WordPress, but it has its tribe. You’ll find thousands of blogs to magazines and even journals using this online publication.

The Ghost Desktop app is quite intuitive and fast due to the simple and more focused style of writing.

The Ghost Desktop application is built on Electron which allows users to focus on writing by providing little things like shortcuts during writing while remaining versatile enough to manage multiple blogs. The Ghost app uses JavaScript and Node.js for delivering an excellent user experience.

Messaging Apps

Some of the world’s most used messaging desktop apps are built on Electron. A good example is Slack Desktop for macOS.

This app is built on the Electron framework and is responsible for its fast performance and unique look compared to what you have in the browser experience.

With the Slack Desktop app, you get to enjoy all the features you experience on the browser version plus better organization and focused workspace to communicate with your team.

Another top messaging app that is based on the Electron framework is Whatsapp Desktop Messenger App which provides more for you while working less. Thanks to the Electron framework, Whatsapp was able to streamline and revolutionize its unique messaging app as well as taking it to the next level.

The electron framework has dominated for a while and based on the number of popular desktop applications using it, we expect it to c

ElectronJS — The Future of Desktop Application was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

The blockchain is arguably one of the most brilliant inventions of the millennium and it has continued to evolve over the years. Blockchain allows for the distribution of digital information, thereby creating the chords that hold the new internet together.

Originally created by a group of people or persons called Satoshi Nakamoto, Blockchain was devised to power the first digital currency, Bitcoin but it has evolved over the years with new meaning and application ascribed to it.

Blockchain is durable

Like the internet, blockchain technology is quite robust and durable and this is made possible by its ability to store blocks of information that are similar across its networks — preventing control by a single entity as well as removing the incidence of a single point of failure.

When you build on the blockchain, the only problems you’ll encounter will be based on bad intentions and human error, not the underlying concepts that hold the technology.

Blockchain and Bitcoin

One of the most popular cases use for blockchain is for bitcoin transactions. When you send bitcoin to someone, the nodes scan the network to validate possession of those currencies and then, once the information is confirmed, the transaction gets added to a “block” which gets added to the previous block in the network and then a chain is formed — hence the name “blockchain”.

There is no single point of failure in the blockchain because transactions can’t be undone or tampered with. This can be viewed as an advantage as well as a disadvantage of using digital currencies compared to traditional financial transactions.

Blockchain: Beyond Cryptocurrency

Blockchain at its fundamental level is a decentralized ledger that keeps records across a peer-to-peer network. It’s the greatest advantage is the ease of transfer of assets without the need to go through a centralized third party.

Other potential application of the blockchain network includes manufacturing, automotive where fractional ownership in autonomous cars could be better managed as well as in healthcare.

For healthcare, the blockchain network can create encrypted health information for patients which could then be shared with multiple providers in a safe and secure way.

There are so many applications to the use of blockchain beyond bitcoin. It can be deployed to track and trace the origin of goods from the farm to the warehouse, where the raw materials come from, verify ownership of intellectual property, and help recover stolen goods in transit.

Blockchain application is endless and can only be limited by our imagination. The technology provides better transparency and visibility to the supply chain and increases the value of products as consumers now have detailed information on where they came from.

Blockchain and Manufacturing

Supply chain, identity, and data management are three key areas that blockchain can work on to revolutionize the manufacturing sector.

As a manufacturer, there are so many things you have to deal with that make transparency and accountability challenges.

These range from logistics of shipping new parts as more manufacturers are starting to shift from the traditional break-fix model to the cost-effective subscription-based model that maximizes product uptime.

This allows the manufacturers to leverage predictive analysis so they are more proactive in their maintenance culture and not just reactionary.

With Blockchain, there can be a better track and trace system put in place to enhance the visibility of this process. Blockchain at the top of its technology allows a manufacturer to see its entire global service supply chain repair parts as they move and ensure repairs are done before damage is done.

Blockchain will also ensure this data can create and maintain a shared and continually reconciled database seamlessly in a decentralized manner.

Blockchain and Food Tech

If we go back to the Spinach Scare of 2006, where consumers were confused about the scarcity of their beloved vegetables and then had to wait for weeks to figure out that the major problems affecting the supply of spinach were the lower elevation and the rain. With blockchain, there will be better visibility and transparency in the food chain making sure that the spinach scare of 2006 would be solved in days rather than weeks.

Today’s consumers want to be more informed about the processes involved in the production of the meals they consume.

Blockchain technology allows for that level of transparency and accountability as a consumer can look up information to trace the path their food goes through from the farm to the warehouse to their local grocery store.

They will also be informed about the ingredients as well as other preservatives that they might not want to consume. With so much spotlight on every step taken, food production will also take an ethical turn in terms of how food is grown, caught, and even slaughtered.

We could also expect new but desirable practices are kept like how animals are hunted and what they do with their skin as well as whether or not fishes are responsibly caught at sea.

Blockchain goes a long way in addressing a lot of issues that we can have and it is a track and trace dream for manufacturers. Adding Blockchain to a manufacturer’s sophisticated but complex system won’t come cheap but once it is put in motion, blockchain technology can make the processes simpler and easier.

Private Blockchain — Joining The Pieces Together was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

It all began when developers wanted to create user-friendly computers with simple layouts. From there, the idea of a graphical user interface (GUI) appeared and it became very easy for anyone to use a PC. There were icons, drop-down menus, and a host of other things that were attractive to the eye with easy description.

UI and UX have come a long way from beautiful screens to pages and buttons. Today, it’s about how well a user interacts with a website, app, or product. UI and UX have characterized how a user can get to point B from point A using the least amount of effort.

Does UI/UX Matter?

In today’s fast-paced society, the most successful products will be the ones that make it easy for the consumer to find what they want as well as those that respond quickly and efficiently to queries.

The internet is now consumer-centric and this is what the whole idea of UI and UX is about. If your product takes too much time to install compared to your competitors, be rest assured that the users will switch to your competitors.

There are a couple of concepts that are important in a user’s experience with a product:

• The processes they go through.
• The sequence of actions they take while interacting with your product.
• The thoughts that arise while using your product
• The final experience or impression they take away.

Every company wants to do one or two things: create new revenue streams, enhance the brand, and build customer engagement. When push comes to shove, there is nothing more important to your business than having an excellent user experience.

The right user experience will make users spend more time using the product, rave about your offerings on social media, and share the experience with their close network.

A picture can change someone’s mood, it can turn a soul. A well-developed user interface and user experience could be the difference between a startup that grows its bottom line and the ones that struggle too.

UI and UX are important. You need to design a product that stands out from the crowd and not just one that is good enough.

User Experience

For your product to stand among the greats, it needs to have a good UX.

This is how you get users to interact with you and some might even provide useful feedback on how you can improve further.

This type of feedback is critical as the users are the ones using the product and they can tell you first hand the areas that need strengthening.

With a laser focus on user experience, your Startup will experience higher revenues and more retention of users. No one enjoys using a product with a poor UX and oftentimes, they find it hard to have loyal users.

Once a user-centric product comes around with a better UX, the users will shift the ground and use the new product.

Companies that focus on user experience lower acquisition costs and fewer complaints. If your user experience is awesome, your best form of marketing will be your customers.

User acceptance and adoption are critical to having a successful product and that is why having a professional development partner like Impulsive Web is important.

How do you accomplish a great user experience?

First, you need to ask yourself some hard questions. You also need to dig deep to understand why customers would abandon the current product in favor of yours.

You also need to find out what the user expects to get from using your product. Whatever their end goal is, your job is to make it as easy as possible for them.

During the testing stage, you should expect feedback from your initial tester to validate whether or not your assumptions were correct. During this stage, be careful not to overwhelm users with too many options that they won’t need or use as well.

User Interface

User Interface is also important to the survival of your product. Your users might tell you they don’t judge a book by its cover, but they do. We all do.

A well-executed design will sell a product before it’s even built. From graphics and animations to a uniform layout and ease of use; the user interface encompasses the credibility and ease of using the product without getting the user confused. You want them to be excited and happy while using your product not disgruntled or pissed off.

This is why we believe it is important you hire a professional development team that can focus on creating visually captivating and easy to use UI.

The user interface in the product is equally as important as what happens in the background for the final goal to be accomplished. This is the first step in designing an efficient and simple navigation interface that can then be tested using a minimum viable product.

It is generally recommended that our Minimum Viable Product should be deployed to a small group of testers (early adopters) for feedback and to gauge interest. MVP is a smart way of building a scaled-down version of the original product.

This saves you the cost of building a product that the customer might not value and it helps you maximize the information learned from the adopters.

At Impulsive Web, we break down every tiny detail and the functionality of the product which shows you what will happen behind the scenes when a particular button is clicked on.

By considering these factors, we can identify what the user interface needs to display for the user and what the product development team needs to do for the user experience to achieve the set goals.

Importance of UX design and how to get it right was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

The amount of time it takes users to complete multiple tasks on your site is affected by the front-end performance. How fast can they click on a button while reading through your product page? How quickly does the search results page take to load? According to Google, if a search results page takes more than half a second to load, it can decrease traffic and your ad revenue. Users expect web pages to load as fast as possible.

Knowing what your customers expect is one part of the equation. The other part is to know if you can cope or not. Designing sites with multiple rich content and lots of dynamic elements means it might be difficult to keep up.

Design for Performance

At the basic level, the performance of a website can be traced to the design. Everything related to performance weighs heavily on the choices made at design. It impacts the load speed and the conversion rate of your site. Every little detail needs to be questioned. What value does the wide background add? Is it causing more havoc than it is helping your site? Sometimes, you might have to do as little as bundling your images into a sprite. This could increase the total homepage size while decreasing the number of requests made. This will lead to a reduction in the entire homepage load time.

Coding for Performance

After designing, coding is another area that affects front-end performance. Sometimes, this could be fixed by cleaning your HTML, and everything else will follow suit. The toughest tasks here will be renaming non-semantic elements in the HTML. Theming concerning semantics like “article” and less with design, you will start reaping the rewards for your efforts. Some of these elements end up in our code due to the need to have more weight in CSS selectors. The right way to go will be to clean up our CSS while adding more specificity but we end up adding IDs that are unnecessary and other elements to our HTML.

Once you’re done with theming semantics together, you should clean up your CSS to eliminate inefficient selectors. Inefficient selectors have a way of increasing page load time, and they are harder to re-design or customize. Efficient CSS selectors are easier to redesign making them quicker to customize the design and styles in the future. Good code is easier to edit, and an editable code leads to good front-end performance.

The third stage is purging your HTML of div-itis. The goal here is to ensure your markup ends up cleaner so the CSS will become smaller and easier to redesign and edit. That way, you end up saving lots of development time and page load time.

Finally, work on creating repurposable code which will also save you time and results when loading your CSS and HTML files. This makes it easier to maintain and redesign your files later and keep them in smaller page sizes.

Optimizing

You have to optimize requests so your site will request fewer ones. Requests occur when the site has to fetch a file or a record. The cleaner the markup, the more infrequent requests the browser has to make, and the less time the user will spend waiting for the browsers to send that information back.

Cleaning the markup is not the destination. JavaScript requests have to be minimized as well, so they only load when it’s necessary. For instance, you don’t have to load a JavaScript file on a mobile design that is only needed for larger screens. You can also configure the site to load JavaScript asynchronously — this ensures the script won’t block any of your content while rendering.

For most sites, a responsive design means more images and CSS files loading, which increases the weight of the requests. You can still get faster load times when the number of requests has been optimized, so only important details are loaded.

Optimizing Images

The first step here is to focus on creating sprites — this ensures you save as many image requests as you can. Creating sprites can reduce the page load time by as much as 10%. After creating sprites, clean up the images by creating one sprite for repeating backgrounds. For no-repeat backgrounds, you can create one transparent sprite for this, inclusive of your logo and icons. At this stage, you will need a tool that will figure out how best to serve your images to your users. After regenerating the images, optimize them by making them smaller with a tool like ImageMagick, TinyPNG and etc.

Measuring Front-End Performance

What is not measured cannot be changed. You have to determine whether your efforts are paying off. This will include using Google’s PageSpeed or YSlow by Yahoo!. These tools will also give you more suggestions on how to improve your page load time, elements blocking faster page rendering, and how to optimize the social scripts on your page.

Front-end performance will always affect your users, and it is your job to understand how it works, measure it, and improve it for a better page load time. A faster front-end performance will lead to better conversion rates for your business, and it will ensure your site accomplish all that it wants to in a short amount of time.

Front End Performance and Enhancements was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

Progressive web apps are no doubt the in-thing in today’s mobile-first business world. This new app has been piped to elicit a paradigm shift in the mobile world.

Even businesses that have invested technological resources in developing local mobile apps are prospecting what it could look like when you have the PWA. These businesses know that the uniqueness of the Power web apps could outshine the longevity of their native apps.

Companies that are yet to devote their resources in a mobile app are now considering the option of making PWA their first resort because they look more economical and relatively easier than the task of developing an entirely new app experience. To put succinctly, all businesses know that the PWA comes with the comparative advantage of a superior web-based app experience that would outshine the features of any native app.

Progressive Web App

The platform-locked design and difficulty to access are some of the restrictions symptomatic of native apps. The PWA which was introduced by Google in 2015 has a unique level of functionality beyond these restrictions.

The distinctive features, therefore, prompted the term progressive web apps. PWA delivers a rare experience incorporating the excellent features found in native apps coupled with the simplicity and unrestricted access to the web.

It is a web-based, app experience. In other words, PWA acts just like a native app but has the merit of being accessible through a mobile web browser.

Thus, the features and functionality are maintained while more users can still easily be connected to the app. By so doing, users are bound to experience functionality better than what is obtained in native apps.

Progressive Web Apps Futuristic Suitability

PWAs tend to outshine mobile apps in the nearest future. It is being suggested that mobile-app are most likely to deliver web-based experiences in the nearest future. Some might remain skeptical of this position going by the number of native apps that mobile users subscribe to on their phones without the web.

PWAs come with novel opportunities that are not peculiar to native apps. Firstly, most mobile phone users are not acquainted with the existence of an app until they access other apps such as the App Store or Google Play. The stressful experience of downloading the app and installing it could be discouraging. Some phone users would not even bother searching for apps at all.

Progressive web apps do not require such a tedious process just to gain access. Users can be connected to app experience via a URL link which makes it easy to share with others.

Phone users would surely savor the experience of merely getting connected to an app through a link.

Accessibility could not have been much easier. Thus, apps could become even more popular than social media platforms when users know that all they require is just to copy and then paste on their friends’ wall. The ability to share progressive web apps will make native apps look outdated and obsolete in no time.

Hybrid Approach

If one is to look at the debate between PWAs and native apps from another facet, it might be rational to conclude that the presence of both types of the app will simply offer a diversity of technological options.

It means mobile companies willing to carve a niche for in their market could offer both the web-based and native versions of the same app.

Companies that have invested in the development of native apps might explore this option in order not to see their investment going down the drain due to a shift in the paradigm.

This hybrid approach flourishes where the two options deviate. Brands can leverage on offering a more comprehensive and all-encompassing experience when they have both versions in their kitty.

PWAs require a lot of power and therefore have a drawback bothering on the lack of performance. Native apps, however, do not have such drawbacks since they do not rely on performance-based features. The hybrid approach will fill the gap left by both versions and deliver an all-around app experience to users.

Thus, companies adopting this hybrid approach could leverage on the accessibility of their power web apps and gain more popularity in the process.

Subscribers can then further urge to obtaining the full native app and enjoy the more comprehensive features. From another shrewd business perspective, PWAs can be exploited for one-time experiences that will create more of a passing craze. For instance, a well-known brand attempting to diversify in its production could release a PWA to aid the market transition and consumer awareness with low marketing overheads being expended.

The native apps will then still retain the basic performance-oriented experiences of the new product.

Why PWAs Might Still Define the Future

App fatigue is one of the maladies affecting mobile users. There seems to be a growing apathy among consumers with many among them not even taking time to check for the latest apps let alone download them.

App stores have tried checking this fatigue by getting rid of old apps that are not being updated or downloaded. There is however still almost 1.5 million apps that are yet to be downloaded.

Most phone users have become frugal with the availability of space on their phone. Many mobile users often feel that they have more than enough apps on their phone which chunk data storage and affects the speed and performance of the phone itself.

PWAs are instantaneous inaccessibility. It does not chunk space like the native apps and therefore has the money to combat app fatigue of mobile phone users.

Users also have the choice of merely saving the PWA’s link if it would not constitute even the most infinitesimal burden. And even if it does, accessing the experience comes by merely entering the URL into a mobile browser.

The flexibility is unique. With Google being the brain behind it, PWA’s are most likely to dictate the tempo in the nearest future.

Progressive Web Apps (PWA) was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

Requirements

One of our clients has a tremendous amount of geo-coordinates from marine traffic across the globe and they wanted to geotag each coordinate for Marine Region, IHO Sea, Country, and Sovereign.

Data Required for each coordinate

Marine Region, IHO Sea, Country, and Sovereign

Conditions

If any of the above data is not available find the nearest data

Problem

Marine Geotagging has several challenges in terms of data availability with the latest and enough information, huge cost involves purchasing Geolocation APIs either from Google or any other service provider and when you have more than 500 Billion coordinates to geotag the problem become more challenging.

A summary of the major problems are as follows:

• More than 500 Billion geo-coordinates
• Not has enough budget to spend on third-party Geolocation services like Google and etc.
• A very limited resource to calculate this amount of data

Third-Party APIs

There are many third-party Geolocation Service APIs available to Geotag GPS coordinates and find the required information. The problem with these APIs are

• They are very slow as geotagging takes time
• Very costly, mostly per coordinate basis
• Not available for all the Marine Regions

These API Services include Google Geolocation Tool, NeutrinoApi, GeoMaker APIs and etc.

The Research

We have started our research keeping two things in our mind

• A super fast and reliable solution
• Data accuracy for at least 100 meters radius

We have divided the problem into two major steps, one is to find a reliable approach with accuracy, test the same on a small chunk of data, and finally scale the same solution for all 500 billion records.

After a week of effort and energy spend in the same direction we find out that the shapefile approach is the only reliable solution that will have the desired accuracy on the data.

The Solution — Part 1

The solution was very simple, find or create a multi-polygon shapefile for Marine Regions including required information like Marine Region name, IHO Sea, Country, and Sovereign.

Match all 500 billion geo-coordinates against this shapefile and check if it exists within any polygon into the shapefile.

If not contains within the polygon, find the nearest polygon to that coordinate.

Shapefile

Finding shapefile on open source was very difficult and we have spent enough time searching on various shape file repositories looking for the exact shapefile, we have found few but was not enough and we have started thinking to create our own using QGis.

Soon we realize that creating an accurate shapefile will take a lot of time and effort and best case scenario if we will be able to create the desired shapefile mapping the same with Marine Region and other required information will be another problem to solve.

Luckily, we have found one shapefile containing 525 polygons with the required information. Intersect_IHO_EEZ_v2_2012.zip available for free http://www.marineregions.org/downloads.php

This file contains information in every polygon

1. MarRegion — Marine Region
2. MARGID — Marine Region Group ID
3. IHO_Sea — IHO Sea information
4. IHO_ID — IHO Sea ID
5. IHO_MARGID — IHO Marine Region Group ID
6. EEZ -
7. EEZ_ID -
8. Country — Country of the region
9. Sovereign — Sovereign of the region
10. Sov_ID — Sovereign ID
11. EEZ_MARGID — EEZ Marine Region Group ID
12. Latitude
13. Longitude
14. Area_m2 — Area of the Region in Meter square

We have imported the shapefile in QGis and it looks like this

Python

Initially, we have tried Python to process the shapefile using the PyShp package and match our test data (GPS coordinates) with the polygons and check if they fall within any of these 525 polygons.

This was a working solution, we were able to match our test records and find the desired information from the shapefile but it was very slow.

The problem was the linear match of the 525 shapes against all the records, we have tested the solution against a smaller data size (10000 geo-coordinates) and it took more than ~14 minutes to process on my machine (MacBook Pro, i7, 16GB RAM, 512GB SSD), we needed a different approach as the data was very huge (500 billion geo-coordinates)

Postgre

We have been using Postgre for a long time and have used in quite a few of our projects as an alternative to MySQL but we have never used Postgre with any plugin.

While researching about the same we find out about PostGIS a Postgre plugin for Geospatial data, we have installed and started looking into the same.

The result was pretty amazing, it was able to process 10000 geo-coordinates in less than ~0.02 seconds as it was using indexes on the 525 shapes we have imported into the PostGIS.

We have decided to dig more into PostGIS to make it even faster to process all 500 billion geo-coordinates at once.

Later, we realized that it is not that easy to process 500 billion records at once and we must divide the same into smaller chunks and it worked.

1. Split the main CSV file containing 500 billion geo-coordinates to 1000 CSV files containing 500 million geo-coordinates each using Python.
2. Written a Postgre function to read the CSV file from disk and import into a Postgre table
3. Generate point using longitude and latitude and store into the same table
4. Match each point against the 525 polygons if they fall within any of them
5. Store the matching ID of the shapes into the same table
6. Populate a new table with matching records along with the geo-coordinates

The Solution — Part 2

The actual implementation, commands, and table and data format are as follows:

Splitting CSV

The actual CSV of all records contains only two columns for longitude and latitude and have more than 500 billion records,

The first step to split the CSV into 1000 CSVs containing only 500 million records each. We have used Python to do the same

# split CSV for every 1 million records
# Filename will be 
csvfile = open('all_records.csv', 'r').readlines()
filename = 1
for i in range(len(csvfile)):
    if i % 1000000 == 0:
      open(str(filename) + '.csv','w+').writelines(csvfile[i:i+1000000])
      filename += 1

After running the above, we have 1000 CSV files containing 1 million records each.

Postgre Import — Shapefile

We have used shp2pgsql command from PostGIS to import shapefile into Postgre

shp2pgsql -I -s   . | psql -U postgres -d ;

Postgre Import — CSV

To import CSV file in Postgre we have used COPY command of Postgre, see the documentation on below link

https://www.postgresql.org/docs/9.2/static/sql-copy.html

COPY lat_long (long, lat) FROM '/path/to/csv-1.csv' WITH DELIMITER ',';

Postgre (PostGIS) — Make Point

We only have longitude and latitude in our records and to match the same with shapes we need a point, so we have used ST_MakePiont command to generate point from the given longitude and latitude

UPDATE oceans.all_records SET geom = ST_MakePoint(long, lat);

See the ST_MakePoint documentation at below link

http://www.postgis.net/docs/ST_MakePoint.html

Postgre — If the point is within the shape

To check if a point is within the shape we have used the ST_Within method of PostGIS

CREATE OR REPLACE FUNCTION GET_SID(p geometry)
RETURNS INTEGER AS $$
ndeclare result INT; 
BEGIN 
SELECT id INTO result
FROM oceans.shapes
WHERE ST_Within(geom, p) IS NOT NULL; 
return result; 
END;
$$ LANGUAGE plpgsql;

UPDATE oceans.all_records SET sid = GET_SID(geom);

We have created a function that takes POINT geometry as input and returns the Shape ID which contains the passed geometry.

Using the UPDATE statement we have updated our all_records table.

Postgre — Nearest shape

There are few records that don’t have shape information as they are not within any of the 525 shapes we have imported, to add the nearest shape into those records we have used ST_Distance method from PostGIS.

CREATE OR REPLACE FUNCTION GET_SID_MIN_DISTANCE(d geometry)
RETURNS INTEGER AS $$ 
declare   distance INT; 
id INT;
BEGIN 
SELECT Min(ST_Distance(s.geom, d)) AS distance, s.id
INTO distance, id
FROM oceans.shapes AS s 
GROUP BY s.gid
ORDER BY distance 
LIMIT 1; 
return sid; 
END;
$$ LANGUAGE plpgsql;

UPDATE oceans.all_records SET sid = GET_SID_MIN_DISTANCE(geom) WHERE sid IS NULL;

We have created a function GET_SID_MIN_DISTANCE which takes POINT geometry and returns the minimum distance shape from the shapes table.

Using UPDATE command we have updated the rest of the SIDs in the all_records table.

After all the above operations we have the all_records table with SID containing Shape ID from shapes table, which can be directly joined to populate the final records including Marine Region, IHO Sea, Country, Sovereign from Shapes and Longitude, Latitude from Records.

SELECT p.lat, p.long, s.marregion, s.iho_sea, s.country, s.sovereign
FROM oceans.all_records AS p
JOIN oceans.shapes AS s
ON p.sid;

Conclusion

By spending time on research we were able to Geotag all the 500 billion geo-coordinates with the matching records without spending a huge amount of money on third-party Geo Tagging services.

When there is a challenge we tend to spend most of the time researching, reading, and testing various available options to find the best fit within the requirements and budget. Sometimes it takes a while to make a decision but knowing the boundary is very important.

We are not saying that spend most of the time in research for everything, there are things which lots of people have already spent the time it required we should leverage their effort and try not to reinvent the wheel but we should double-check the environment that specific solution is designed or developed.

Let us know if have faced similar challenges, we would be really happy to know how you have overcome and what you have built.

Cheers!

Marine Geotagging was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background

Today the key feature of any website is how fast it performs and how fast we can get the results out of it as today nobody has time to wait and stare the screen to get the desired output.

To fix that we need Performance Testing and to do performance testing we need scripts, to get the scripts we need time and a performance engineer which is a time-consuming process.

In performance testing, the most important thing is to identify the scenarios for performance, the scripts, and then to identify and analyze the results.

The most time taking and painful part in whole is the scripting so what if I say we made that easy!!

Just one click and your scripts are ready to run on any platform you need, be it LoadRunner, JMeter, Neoload, etc.

The Problem

Today almost every tool has the capability of recording the steps you perform on the AUT and then generating a script out of it, but the major challenge is to correlate that.

So for a nonperformance guy, the question will be “What does this correlation mean?”

Explaining that in a very layman terms is capturing the dynamic values returned by the server and then sending that with every request so that you can fool your server that the request is valid and you should get a valid response.

Now, this might seem to be a very small and easy job but let me tell you it wasn’t.

We need to dive deep into every response, rerun the scripts to find out the difference, looking into those dynamic ids and then keep correlating them with the help of regular expressions which takes a hell lot of time and effort.

The Solution

We initially started with a thought “what if this can be automated?” as there are many who are working on automated reporting automated load models etc but no tool has this capability to automate the core, The Scripting!!

Then the biggest question was Maybe this is not achievable because if it is then there are tech giants who would have done this, no?

But apart all we thought of giving it a try and see we get something else it will be a lesson learned.

We researched a lot and then found out a way to proceed on:

• Get the HAR files of each action performed from chrome.
• Repeat the steps performed and get another HAR to compare.
• Compare the HAR’s and get dynamic values and boundaries.
• Correlate them by identifying the left and right boundaries.
• Put that correlation in place of the tags.
• Finally, generate a script in the desired format as per the tool and get the script ready.

Getting the HAR File

To get the HAR we also made a recorder that can record the steps we perform and then re-run those which helped us remove one more human intervention.

The first challenge we faced was to generate the HAR and we tried Phantom JS and puppeteer but later learned that it does not have the right capabilities to create multiple HAR then we researched a bit more and finally settled on selenium with JAVA. One problem solved!!

Comparing the HAR Files

The next was to compare 2 HAR’s and get the differences, easy we did that in a while by storing each element as a key-value pair and later comparing them to get the results.

Backtracking

Now the most complicated process was to correlate them by identifying the left and right boundaries on the fly and get the values correlated. We struggled a lot in that and finally figured out that every application responds differently so we cannot automate everything at a go.

We figured out some very basic and the most common values returned which should be taken care of first and we started covering them.

<(anything) key = value (anything)/> 
<(anything) key (anything) value (anything)>
<(anything) value (anything) key (anything)>

To get the values in Header, cookies, and Body

Finding the dynamic boundaries was really tough but after a lot of iterations and struggle we did that finally

Now the next challenge was to put those pieces together and create a script out of that. For this, we targeted JMeter script first and created a valid JMX out of that which can be easily run in Jmeter.

We put all the pieces together and created a tool which can just record the actions you do on the browser as a plugin and send the HAR generated to our server where our engine works and create 2 runs of the project to compare, the engine compares then and auto-generates a JMX file out of this which we can run at our servers and we will get beautiful reports out of that.

Conclusion

We are still enhancing our tool each and every day by adding more and more conditions in that and making the system more intelligent.

We research and try to challenge our skills and sometimes we achieve which seems impossible.

Not believing in the story? Ask a demo for the product to see it live!!

Automated performance testing with JMeter was originally published in Impulsive Web on Medium, where people are continuing the conversation by highlighting and responding to this story.