The Origin

Git came out of a need. As the Linux kernal was being developed, there were three big problems which needed to be addressed: large amount of contributors, the source code was getting bigger, and the need for an easier way to manage the source code.

There were three usability design goals behind git:

• Support distributed workflows
• Offer safeguards
• High performance

This led the designers to build git using DAGs. A DAG is a directed acyclic graph. This design is very important to git as it is used to store content, reference pointers for heads, object model representation, and remote protocols.

Version Control System Design

There are three functions for a Version Control System(VCS):

• Store content
• Track content changes
• Disturb content and history with collaborators

Content Storage

There are two design choices for storing content in VCS: delta-based changeset or a directed acyclic graph(DAG). A DAG has objects forming a hierarchy to mirror the content filesystem tree which gives a snapshot of the commit. Git stores content as a DAG using different types of objects. The file: blob.c — is how information is stored in git. The code written above is from the file: blob.c. The information is stored as an object and uses an sha1 to validate the blob.

The blob is then stored in a tree. This tree is shown at tree.c. There are multiple read_one_entry functions, which are used to read the blobs and store them into trees. There are also two read_tree functions. One is implemented with recursion and another is implemented iteratively. These two files in combination with each other are how files are stored in a repository.

Commit and Merge Histories

VCS software usually uses one of the following approaches to track changes:

• Linear History
• Directed Acyclic graph for history

Git uses DAGs to store a project’s history. Each commit contains metadata regarding its ancestors. It also has the ability to directly support branching to record most merge history cases. Having a DAG storing the contents, allows Git to:

• have nodes with the same identity as those in many different commits
• allow nodes to merge the their contents

Distribution

The three ways VCS offers methods of storing and distributing data, which include:

• Local-only: all source code is stored locally
• Central Server: one specific repository used to store all changes to the repository
• Distributed model: Each individual has their own local workspace where each person works, and then pushes commits to a remote repository.

Git follows a distributed model in conjunction with a central server to store changes. When storing changes for Git, all changes are first stored locally. and then the pending commits are pushed to a public repository. The public repository is used by all individual contributors. For the local commit, the local Git repository creates a new object to represent the file that was changed.

When a new file is created, a new tree object is added. A DAG is then created starting from the newly created tree objects which then point to blobs, and reference the newly created blob. The two levels of added complexity allow for greater flexibility for the workflow and publishing capabilities.

Note: A blob represents a file stored in the repository

The Toolkit

The Github toolkit includes many command line and UI tools on various operating systems. (There is even a Github for Windows project started by the open source community). The majority of the toolkit was written to be very similar to the Unix tradition of the command line.

The toolkit is divided into two parts: the plumbing and the porcelain. The plumbing consists of low-level commands to enable content tracking and DAG manipulation. The porcelain is a subset of git commands used to maintain repositories and communicate between repositories for collaboration.

The Repo

To initialize a git repository or to turn an existing local directory to a Git repository, the user issues a git init command. This command creates a .git sub directory, which consists of the following files:

The .git repositories consists of the following sub directories

• Configuration: the .git/config, .git/description, and .git/info/exclude files are used to configure the local repository.
• Hooks: the .git/hooks directory contains scripts that run during certain life cycle events
• Staging Area: the .git/index file is the staging area for the working directory
• Object Database: the objects directory is the default Git object database. It contains all contents and pointers to local content.
• References: the refs directory is the default location to store reference pointers for both local and remote repositories, tags and heads.

The .git directory is the actual directory which contains a working set of files. the directory which is the parent to the .git directory, consists of the local repository files. An empty git repository can be initialized using the command, git init — bare. This creates a git repository without a working directory

The Git index: git/index, is another important file which provides the staging area between local and remote repositories. It stages specific changes within one file. Specific changes to a file can be added using: git add -p.

In addition to the files in the .git directory, it is important to understand what happens when some of these commands are executed.

• git checkout [branch]: moves the HEAD reference of the local repository to branch reference path, populate the index with the head data and refresh the working directory.
• git add [files]: cross references the checksums of the files with the corresponding entries in the Git index to see if the files in the remote repository need to be updated.

The Object Database

Every object in the local repository is built around four basic primitive objects. These object types are:

• Tree: an element in a tree can be another tree or a blob
• Blob: a file stored in the repository
• Commit: a commit points to a tree representing the top level directory for that commit and parent commits
• Tag: a tag has a name and points to a commit at the point in the repository history that the tag represents

A SHA (a 40-digit object identity) is used to reference primitive objects. If two objects are identical, then they will have the same SHA. The properties of the SHA guarantee the identity of the object and help to enable some safeguards against corruption.

Storage and Compression Techniques

Git handles storage by packing objects in a compressed format, and uses an index file to locate specific objects in the corresponding packed file. The command git count-objects is used to count the number of loose objects in a local Git repository.

The packet file format evolved. The initial format stores CRC checksums for the pack file and the index file in the index file itself. Git establishes remote communications by calculating the commits and contents that need to be sent over the wire to synchronize repositories. It also generates the pack file format on the fly to send back using the client’s protocol.

Merge Histories

Git handles merge history by storing the previous versions of the code using a DAG. The DAG allows git to know where on the current branch the changes are made. By having the changes on the head of the branch, git is able to see where the commits are being made.

Final Thoughts

The trade-offs and decisions that went into making git have led it to be one of the more successful VCS. It’s flexibility and implementation allows for many people to work on the same project. It has addressed and solved a majority of the problems the Linux Kernel developers were having with managing the kernel source code. There were definitely a lot of lessons learned while exploring git’s design.

Ignoring security could be one of the biggest mistakes a business or a company makes. Not only does it hurt the user, but it negatively affects the business, damaging its reputations and exposing vulnerabilities. Regardless of what is being developed, all developers should consider security in their design. They should explore the tradeoff between usability vs. security, and the access points for the requested information.

Ashley Madison

Just recently AshleyMadison.com, a site people visit to have affairs was recently hacked. The hackers stole personal information from Avid Life Media (ALM), which stores information regarding users of various dating websites. In addition to account data from 40 million users, the hackers also leaked maps of internal company servers, employee account information, and company bank account data and salary information. The site claimed to have a “full delete” feature which erases all user information even from the backend servers for 20 dollars. However user information such as credit card information, and even personal details were found. This leak poses a threat to the customers and the site as hackers are threatening to release user information if the CEO does not take the site down.

Read more at: http://krebsonsecurity.com/2015/07/online-cheating-site-ashleymadison-hacked/

Repercussions of ignoring security

This type of hack poses a lot of risk to the business including:

• destroying the service’s reputation
• Risk user lives by sacrificing confidential information
• Decrease in revenue

Regardless of the business, security is important. The hackers are now threatening to release critical user information until the site owner shuts itself down. The business and its leaders now have to make a critical decision whether they lose their business or the site’s users lives are now endanger.

Venmo

Venmo is a mobile application used by individuals to transfer payments. A professional poker player, Mohsin Charania was hacked for $2,000 in 2014. Another individual, Brant Trinh, lost $3,000 from Venmo.

Mobile applications are also endanger of not being fully secure. Mobile banking apps are new. There are many challenges developing mobile banking apps. All these apps require money to be transferred wirelessly. Any information which is transferred wirelessly is susceptible to being hijacked.

Read more at: http://nextshark.com/venmo-hacked/#rmns

Consider Security for Development

A security program must protect the confidentiality, integrity, and avalability of the data.

• Confidentiality: allowing only authorized users to view the neccessary peices of information.
• Integrity: Ensure that only authorized user can modify the data.
• Availability: provide uninterrupted access by authorized users to important computer resources and data

Confidentiality, integrity, and availability are the three core tenants to consider for security reasons. The designer has to be aware of the threats which compromise security, the risks associated with the threats, the cost of implementing the proper countermeasures, and finally a cost-benefit analysis to determine if it is worth implementing those countermeasures. The cost of implementing these security measures can really take a toll on the business, but is a neccessity to ensure the user’s safety. Making the correct trade-offs between the two really helps to identify flaws and vulnerabilities. It also identifies which assets are more important and helps developers prioritize their needs.

Now What to do When You’re hacked

Regardless of how much developers or businesses prepare, if someone wants to hack into the system, they will find a way. There is no stopping these individuals from stealing user data. It is important to recognize this and what to do when critical data is stolen. Having a mitigation plan or a backup plan can really help deal with these attacks. The backup plan will change depending on the business but dealing with these attacks and having a plan when getting hacked is very important.

Conclusion

As shown above through the recent hacks and attacks on networks, security is one of the most important concepts needed when developing a product. When a person users a service or website, there is a trust established. This trust is broken when there is a data breach and users lose their information as they cannot trust or rely on the service anymore. As important as it is to release a product quickly to the market, it is more important to have a way to deal with being hacked and making it as difficult as possible for hackers.

Note: As always, I really appreciate any comments or any suggestions as to how to improve upon what I write. So if you like what I wrote and feel it can be better please feel free to leave a comment.

Basics

When building a website, it is always important to consider system architecture. The architecture for a system could be what makes or breaks the site itself. The backend of a website has many responsibilities including:

• Hosting the webpage
• Storing user and website data
• Handling user requests and service APIs

Therefore when designing a system architecture there are some very important components to consider. There are many core factors that need to be considered when building this system. These include:

• Services
• Redundancy
• Partitions
• Failure Handling

All four of these factors lead to different type of architecture. When one is favored over the other, we have a different type of system architecture which would be better for a website that serves a different purpose. For example, a shared-nothing web architecture may be more useful for an emergency services provider than a service-based architecture. All of these factors are important when designing an architecture to support different web processes. Making the right tradeoffs between those four factors can make or break a website. Learning to prioritize one over the other will lead to good system design concepts.

Services

Services are a vital part of system architecture. Some systems that are designed with each service as its own defined interface. Each service has it’s own distinct functional context, and interaction with anything outside of that context takes place through an abstract interface. By breaking down the system based on service, it defines the relationships between the service, underlying environment, and it’s customers. This enables individual components and services to scale independently and allows for easier debugging of a service when it goes down.

When adding services it is important to remember to consider the following principles:

• Determine the system’s needs
• Benchmark alternatives
• Understand system failures
• Plan for Failures

Systems design where each service has its own interface is known as a Service-Oriented Architecture (SOA).

“ A service-oriented architecture (SOA) is an architectural pattern in computer software design in which application components provide services to other components via a communications protocol, typically over a network. The principles of service-orientation are independent of any vendor, product or technology. “ — Chapter 1: Service Oriented Architecture (SOA)

A SOA helps to identify business needs much easier as services which are used more ofthen will be prioritzed and have the architecture revolve around them.

Redundancy

Redundancy is important to consider when to gracefully handle failures. Having multiple copies or different versions compensates for the failure of a single node. Redundancy removes single points of failure and provides backups or spare functionality in a crisis.

When redundancy becomes the most important aspect of the system architecture, we would consider a shared-nothing architecture, where each node operates independently. This helps with scalability as new nodes are added without special conditions or knowledge. In a shared-nothing architecture each node or service is independent and has no knowledge of the components of the other nodes or services in the system. The biggest advantage of this type of architecture is that it allows for no one single point of failure which can shut down the entire system.

Partitions

Partitions enable horizontal and vertical scaling. Both concepts allow systems to expand and scale based on user needs and demands. It allows for easier management of system components and helps with better understanding the different components.

Vertical scaling means adding more resources to an individual server. This is accomplished though having the individual resources handle more data and services then it normally could on it’s own.

Horizontal scaling is adding more nodes to a an older system. This is an intrinsic principle for system architecture. Partitions allow for this to be implemented in practice. These partitions can be distributed based on different factors such as: functionality, geographic boundaries, or other criterias.

In this partition + redundancy based architecture there are multiple servers used to store data and files. Data is duplicated across servers which allows different components quicker access to the resources it would need. These also allows for easier data recovery. When a certain piece of data is lost, it can be recovered from another server. These are two very big benefits to this sort of architecture, however modifying the original data becomes a challenge since there are now multiple points where the data must be modified.

To help with the challenge, an ID or unique key could be assigned to the specfic sets of data. This allows for easier tracking of files and helps to keep track of the different areas where the data needs to be modifed or updated.

The two key issues with a partition + redundancy based architecture are:

• data locality
• inconsistency.

For data locality, the location of the partition to the operation point determines system performance. The further away the data, the higher the cost for fetching that information.

Inconsistency issues arise because ther could be a chance for race conditions when adding multiple resources. This could prevent the data the user is viewing or using from being accurate.

The two issues written above should be taken into account for system architecture design. However this sort of architecture is important when creating a system to be scalable.

The Building Blocks of Fast and Scalable Data Access

The two biggest challenges for scalable application design are:

• scaling access to the app
• scaling access to the database

There are multiple methods to handle these challenges including but not limited to:

• Caches
• Proxies
• Indexes
• Load Balacing
• Queues

These are some approaches that could be used to solve problems with scalability. It also allows for systems to respond quicker to user requests and solve multiple problems when dealing with different features.

Caches

Caches take advantage of The Locality of Reference Principle. A cache is limited in space, but allows for quicker access to data than the original data source. Caches hold the most recently accessed items. These can be placed acrossed different parts of a system architecture, and depending on the placement of these caches, the chances of a cache miss could either decrease or increase.The two methods for dealing with this hurdle are: global caches and distributed caches.

Global Caches: A single space dedicated for cache space used by all the nodes in a graph. A global cache requires either a simple server, or file store accessible by the request layer nodes. Either the cache or the nodes are responsible for data retreival.

Distributed Caches: In a distributed cache, each node is responsible for its own cache to store data. This is usually implemented with a hashing function, such that each node can use that function to quickly look up in the cache if the data is avaliable.

Proxies

A proxy is an “intermediate piece of hardware/software that receives requests from clients and relays them to the backend origin servers.” There are several uses for proxies which can help improve the system performance, including:

1. collapse forwarding : speed up data access by collapsing the same requests into one request.
2. collapse requests : collapse data that is located on similar data partitions or location.

Placing a cache in front of a proxy allows even quicker access to data that is infront of a partition. The cache can retrieve data quickly from memory, therfore it handles multiple requests for the same results.

TODO: Insert image depicting using a proxy to collapse requests for data

Indexes

Indexes allow data to be quickly accessed. It trades extra storage overhead and slower writes for faster reads. Optimizing indexes depends on the users. Creating indexes for data which is more likely to be used will allow for better system performance. Indexes are required to optimize data access for large dat sets with a small payload. It is difficult to find a small payload in a large data set, and indexes solve this problem.

Indexes can also be used like a table of contents. It could direct users to the data location. These indexes are stored in memory close to the incoming client request.

There are also often different layers of indexes that serve as a map moving people along until they reach a specific piece of data. These intermediate indexes are similar to indexes that lead to the data the user wants, however instead of containing the data location, it contains the next index or location.

Intermediate indexes and representing the data in smaller sections allows for better solutions to big data problems. Even with the data spread through the different servers, it could be easily accesed with index lookups.

Load Balancers

Load balancers are very critical in building a distributed system. They distribute the load across nodes responsible for servicing requests. Load balancers handle a lot of simultanious connections and route those connects to the appropriate request nodes, allowing the system to scale to services by only having to adding nodes.

Load balancers are an easy way to expand system capacity and play an essential role in distributed system architecture.

Queues

While load balancers, partitions, and indexes allow for quicker writes, they don’t address the problem of needing to read data quickly. Systems where writes or any other task may take a long time, queues are a common way to achieve better performance and availability. Having a queue allows for asynchronous transactions. They allow for tasks to be handled in the order they are received.

When a task comes in, its added to the queue and then the next avaliable service picks it up. A queue also provides protection from service outages and failures. It enforces a QoS (Quality of Service) which guarantees to expose clients to intermittent service outages. Queues are neccessary to handle distributed communication between the various parts of a large-scale distributed system

Conclusion

Designing efficient systems is very difficult. There are many factors which need to be considered. System design requires a balance between the four factors: services, redundancy, partitions, and handling failure. A system that holds a large amount of data may favor having partitions instead of providing many services. While a system that requires a large amount of user interactions may have a ton of services to process user requests. Caches, Indexes, Queues and Load Balances help to deal with the trade-offs. They balance what is traded away by providing an alternative way to deal with the system level trade-off. A lot of these factors were taken into consideration when designing various system architectures which support a large amount of websites we use today such as Facebook, and Amazon.

Please feel free to leave any comments, suggestions, or anything that I might have mentioned which is currently outdated. I would really like to learn from other people and appreciate anything others may say.

Recently I started to read a book called: The Architecture Of Open Source Systems. Overall I am really enjoying this book. It talks about a lot of different open source systems and the brains behind these systems. The first topic I would like to explore is scalable web architecture and distributed systems.

When designing large web systems and some distributed systems, there are certain building blocks that need to be taken into account. These are some of the tradeoffs that need to be taken into account when creating a large web system.

Principles of Web Distributed Systems Design

A web site is basically a utility which allows users to connect with remote resources via the Internet. A scalable web system just means that the resources or access to the resources are distributed across multiple servers instead of just one. There are some key principles that influence the design of a large-scale web systems that are listed below:

• Availability: the uptime of a website is very critical to a website’s reputation and functionality. For some companies it can really make or break them. A online retail website such as Amazon will need to constantly be running and users should always be able to access the resources and services from Amazon.
• Performance: website performance is one principle that is very important for most sites. For example, the speed and validity of the results from a search engine will dictate if users want to use this website. Creating a system that is optimized for fast responses and low latency is a key to success.
• Reliability: A system needs to be reliable. The request for data needs to be consistent. If there is a change in the data, the user needs to be know about to change.
• Scalability: This is a very broad but important aspect. It can refer to how much more storage the servers can handle or even how much more traffic the system can work with. Scalability is important to consider as it helps to understand how much traffic the website can handle and how it handles large amounts of data.
• Manageability: While designing a system, it is important to consider how the system will be managed. The operation, maintenance and update costs, as well as how easy it is to operate the system. It is important to take these factors into consideration when managing a system.
• Cost: Of course the hardware and software costs need to be considered, but there are other items which are associated with a cost. This includes what is needed to deploy and maintain the system, how many developers and training and what else needs to be used to not only develop but deploy the system.

So overall it is important to consider all these factors when designing a large web application. Some of these factors might also not seem important right away, but later on will come back to haunt you. Imagine if Amazon ignored Scalability, the site would not be as it is today. It would really hurt their business. Or what if Medium did not consider Reliability. That would really hurt what we are writing on the site. Would the posts we view be what we expect?

These provide a baseline for what needs to be considered when we establish a distributed web architecture.