<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sergey Meshkov on Medium]]></title>
        <description><![CDATA[Stories by Sergey Meshkov on Medium]]></description>
        <link>https://medium.com/@sergmesh?source=rss-10e05f301d16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*F9Wyw2E53J4rro6h.jpg</url>
            <title>Stories by Sergey Meshkov on Medium</title>
            <link>https://medium.com/@sergmesh?source=rss-10e05f301d16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:44:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sergmesh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Notes On Data — Maintenance]]></title>
            <link>https://medium.com/@sergmesh/notes-on-data-maintenance-2783b57200cd?source=rss-10e05f301d16------2</link>
            <guid isPermaLink="false">https://medium.com/p/2783b57200cd</guid>
            <dc:creator><![CDATA[Sergey Meshkov]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 22:46:47 GMT</pubDate>
            <atom:updated>2025-09-15T22:46:47.940Z</atom:updated>
            <content:encoded><![CDATA[<h3>Notes On Data — Maintenance</h3><p><em>A series of remarks for a data architect to pay attention to when designing data management systems.</em></p><p>This section is about small and annoying tasks we can not forget when designing data processing solutions.</p><h4>Size</h4><p>Size is what’s killing all data processing systems. Let’s imagine a scenario: <br>A new system works great and everyone is excited about it. Clients are happy and everything seems to be fine but there is a ticking bomb under the system. The size of the data on the disk is manageable, but it is growing. <br>Some time later the database is not able to accept new data anymore and the system is in an unmaintainable state. The database is so huge that DBAs are not able to do a backup of it or a query time is so big that the clients do not receive a response in time. <br>We can see multiple scenarios like the one above in real life, just not in such a dramatic way.</p><blockquote>The size is one of the critical parameters that any requirements must have specified at the very beginning of a new project.</blockquote><p>When architects plan hardware and choose technologies to use, they need to think about Service Level Agreements (SLA) and how to fulfil them. Everything comes from the requirements and those requirements are being propagated down to developers and hardware support specialists. <br>To avoid any “unexpected” situations there has to be a monitoring solution checking how much space is used for the data a system manages.<br>Regular (usually every 3 months) review of the space usage should be performed. If required, a new storage should be added to avoid all space to be consumed.</p><blockquote>A system needs to maintain constant data size throughout a system lifetime.</blockquote><h4>Archival Process</h4><p>Depending on the SLA in some cases it is possible to offload data not required for immediate access from the main storage to a secondary storage solution. For example, application logging data is stored in a such way. Actual logs are on a file system and archived logs are somewhere on the Amazon S3 object storage.<br>There has to be an automatic process that moves processed data from the main storage to a secondary one. This will keep the size of the data at a constant level. Archiving can not be a manual process, as people go on vacation, get sick or simply forget about routine tasks.<br>Storage for the archived data is potentially growing all the time. So the solution must be cheap enough to accumulate about 10 years of data without going bankrupt. For example, <a href="https://aws.amazon.com/s3/">Amazon S3</a> + <a href="https://iceberg.apache.org/">Apache Iceberg</a> allows the storage of a lot of data at the same time providing decent query time.<br>Archival data is a read only data and should not be changed as you can not change the past. <a href="https://parquet.apache.org/">Parquet</a> file data format suits well for this kind of task.</p><h4>Backup</h4><p>Backup guarantees a system will be available after a total system crash. The procedures are specific to the main storage solution used. Sometimes clients tend to have 2 environments working in parallel in production.<br>This obviously increases the availability of the system, but it doubles the space required.<br>In spite of many ways to define a backup policy, one thing to remember is that a backup policy affects the space required for the overall solution.</p><blockquote>overall space required = main app size + archived data size + backup size</blockquote><h4>Modifications</h4><p>Data is split into operational (frequently used and modified) and archived. Archived data is read only data. This fact gives a nice possibility to save historical data into read only data formats, like parquet files. On the other hand, operational data could be modified. So, we can concentrate on solving modification issues just for operational data.<br>It’s good to think about data as a slow moving syrup. The closer it is to the “hot” source the faster it flows, and the further it moves from the source the cooler it becomes and moves slower.<br>Similar kind of thinking could be applied to the data. Operational data is hot and being updated constantly and with historical data it is the opposite. The further from the “hot” spot the more “read only” it becomes.<br>So, your operational data could be stored in the database and archived data could be on S3 + Apache Iceberg + Apache Parquet files.<br>You can have as many stages as you want, for example you can have two databases, one for hot data and one for historic, but the availability of the historic database will be less than the hot one.<br>It’s common to represent storage solutions with “tiers”, being tier 0 as a most available data store going to tier 1, 2 and so on depending on the requirements.</p><h4>Synchronization</h4><p>Quite a popular solution for operational storage is to use database or NoSQL caches. It is important to know that in a distributed system data locking should be applied to guarantee correctness.</p><blockquote>The data must be correct at any stage of processing.</blockquote><p>If you think that you have a database with transactions, and it will prevent concurrent modifications issues, then you are only partially correct. Transactions support is not enough to guarantee data correctness in a distributed system. If two independent processes are modifying a record in a database then synchronization must be applied.<br>Think about two parallel read — modify — write cycles applied to the same record. So, in this case pessimistic or optimistic locking must be applied to guarantee correctness.<br>Pessimistic transactions at the database could be done by running SELECT FOR UPDATE commands and optimistic locking could be implemented by data versioning.</p><p>The main idea of the optimistic locking is you basically have a version attached to the data entry. Every time some component modifies a data entry the version must be incremented. Each component carries a version with its modification request and if the versions don’t match a reread should be performed.<br>Just this small addition to the data entry will save you a lot of time when debugging a distributed system.</p><h4>Serialization</h4><p>Is an XML format good for storing data? Is JSON better than XML? How to answer a question if a serialization format good or not?</p><blockquote>Serialization is a process of moving data from in-memory representation in to a byte array.</blockquote><p>In-memory representation is feature reach. It has abstraction/model layer applied to the raw data. So, additional information should be stored along with the actual information.<br>For example, record with a field “price” has a value 125. You can’t simply store 125 as the data context is not present. Information about the field must be stored in the byte array along the actual data to be able to read the information back and move the raw data from a byte array into the in-memory object representation.</p><blockquote>One of the key parameters is the ratio of actual data and the fully serialized representation.</blockquote><p>For example:</p><pre>{&quot;firstName&quot;:&quot;John&quot;, &quot;lastName&quot;:&quot;Smith&quot;}</pre><pre>&lt;?xml version=&quot;1.0&quot;?&gt;<br>&lt;person&gt;<br>&lt;firstName&gt;John&lt;/firstName&gt;&lt;lastName&gt;Smith&lt;/lastName&gt;<br>&lt;/person&gt;</pre><pre>JohnSmith</pre><p>JSON: 40 bytes<br>XML: 91 bytes<br>Actual Data: 9 bytes</p><p>JSON Rate — 9/40=0.225–77% of the information is for the context<br>XML Rate — 9/91=0.098–90% of the information is for the context</p><p>With this example you can see that JSON is less verbose which will allow you to save money on the disk space. The numbers above are just for the demonstrating of the idea on how to calculate the rate. In each particular case the “typical” data example (or several) should be taken into consideration to calculate the rate. Usually binary data formats are less verbose and more compact which allows saving the space. <br>One of the ways to save space on the context is to just have a reference to a schema stored along with the raw data.</p><p>In some cases people tend to include context information as a part of the serialization format. This will be a massive waste of space. One of the clients I worked for had an idea of storing data in a human-readable json format. To make each record “human-readable” excessive information has been included as part of the serialization to each and every record. As a result, 80% of the storage was wasted.<br>To my question: ’Has anyone looked at the raw data?’ the answer was no.<br>I’m showing this case just an example of what data architects need to think about when designing data storage systems.</p><p>Another non-trivial idea to understand is whether I need to deserialize the entire entry to get a field required from the stored object. In other words, let’s say we have an object in memory, it has some fields, and we managed to serialize it to byte array. Does the serialized format allow to get a certain field without deserializing the entire entry?</p><p>For the sake of example, we serialized data into the XML format and saved it to a database. To get a value from xml I need to read the entire XML into the memory first and then run xpath to extract a field value. On the other hand, we can write a record into a table with multiple columns.<br>So, in the first case we just read the entire XML payload for extracting one field wasting the rest of the document. And when we have multiple columns we can get a value just from one of the columns.</p><p>Formats that allow direct fields access:<br>FlatBuffers (<a href="https://flatbuffers.dev/tutorial/">https://flatbuffers.dev/tutorial/</a>).<br>Oracle Coherence POF format (<a href="https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-applications/using-portable-object-format.html">https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-applications/using-portable-object-format.html</a>) by using PofExtractor (<a href="https://docs.oracle.com/middleware/1213/coherence/java-reference/com/tangosol/util/extractor/PofExtractor.html">https://docs.oracle.com/middleware/1213/coherence/java-reference/com/tangosol/util/extractor/PofExtractor.html</a>).</p><p>So, if we, for the nice idea of “saving space on the disk” implemented zip archiving for the operational data we simply killed CPU. As the only thing our system will do once implemented will be archiving / unarchiving the data.</p><h4>History</h4><p>Over the time the data structure might change and instead of two fields to save we will have to save three. The data structure will change. It will definitely happen as the requirements for the system evolve over time.<br>So, we have to have some kind of data structure versioning as we will have to support multiple versions going forward.<br>The change will not happen instantly, and we will have to support old and new versions for a period of time. Plus, we will have to support reading of the historical data. So, this means that we will have to implement a reader/writer supporting multiple data structures at the same time.</p><blockquote>The data structure version must be stored as part of the serialized payload.</blockquote><p>It can be just a data structure number or a schema reference, but we have to have this information stored as part of the serialized data to be able to read it back from the disk correctly.</p><h4>Data Locality</h4><p>The simplest way to think about data locality is to imagine a multinational website which has dynamic and static content. Static content can be easily offloaded to a local Content Delivery Network (CDN) which will speed up website rendering. Akamai (<a href="https://www.akamai.com/">https://www.akamai.com/</a>) is an example of such CDN.</p><p>Now let’s look a bit deeper. What if all the data we need for a task is local to a process that will run the task? We prepare all required data upfront and give it to the task. So, the task does not need to do any remote calls.<br>This will significantly improve processing speed and throughput.<br>Oracle Coherence (<a href="https://www.oracle.com/uk/java/coherence/">https://www.oracle.com/uk/java/coherence/</a>) is one of the solutions that can help implementing such requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2783b57200cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notes On Data — Security]]></title>
            <link>https://medium.com/@sergmesh/notes-on-data-security-e48e34787f82?source=rss-10e05f301d16------2</link>
            <guid isPermaLink="false">https://medium.com/p/e48e34787f82</guid>
            <dc:creator><![CDATA[Sergey Meshkov]]></dc:creator>
            <pubDate>Sat, 06 Sep 2025 19:20:37 GMT</pubDate>
            <atom:updated>2025-09-06T19:20:37.030Z</atom:updated>
            <content:encoded><![CDATA[<h3>Notes On Data — Security</h3><p><em>A series of remarks for a data architect to pay attention to when designing data management systems.</em></p><p>Data security is the most complex task to implement right. In this article we will talk about topics to remember about when implementing a data storage solutions.</p><p>Let’s start with roles and permissions. There is a Role Based Access Control (RBAC) model which explains how a model should be implemented.<br>NIST Standard (<a href="https://csrc.nist.gov/csrc/media/projects/role-based-access-control/documents/rbac-std-draft.pdf">https://csrc.nist.gov/csrc/media/projects/role-based-access-control/documents/rbac-std-draft.pdf</a>) explains the model and ways to implement it within an organisation.<br>The most interesting part of it is the diagram below which highlights basic elements of the model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4ayIclKSQRkZGypjaTN4Q.png" /></figure><p>As you can see the important part of the model is to define permissions on the data a storage system is going to manage.</p><blockquote>A permission is a combination of operations (read,write,etc.) and objects.</blockquote><p>If operations are more or less clear the objects could have a variety of forms starting from image or pdf document and finishing with a record in a database.</p><blockquote>Objects could be of any nature.</blockquote><p>If we talk about table / record data structure we can define the object as a record. Examples of record based data structure include: database table, csv file, excel, parquet file. The main question a security system should answer is if a person can see this record or not. Record level access allows us to provide view to the entire record with all fields exposed.</p><p>However, a more flexible approach is to have columns (or record’s fields) defined as objects for permissions as well. So, one user can see just an address and another one can see a first / last name only. At first, this approach does not have any sense but once you start thinking about a user of the system as a service that constructs an email salutation header and does not need to know about address of a client, you will realise that just simple separation of fields into different roles will increase the security level of a system significantly. This means that if a hacker hijacks a service he will not be able to get access to the rest of the client’s data.</p><blockquote>By setting up roles and permissions correctly one can achieve a high security level.</blockquote><p>Another important point about permissions is the ability to integrate them into an organisation’s existing security system. Objects and actions are clearly defined at a data level. By nature, permissions are “local” to the data storage system and can be managed by a support team. The next step is to integrate those into a organisation’s security system.</p><p>Pretty much every organisation today has a user management systems with roles assigned to the staff members. From a system’s perspective a role is just a string, there is no link to data permissions that the storage system can handle. The naive approach is to hardcode the organization’s role name into the system. However, it will not be flexible. If tomorrow the management will ask about additional permissions for a role it will be much easier to assign additional permissions to the role provided. So, basically role to permissions association serves as integration point between the actual data and organisation wide authentication control.</p><p>It is important to separate permissions that allow access to a storage system and data level permissions that are specific to a user. Two users with different data level permissions for the same query will see different data. Oracle Label Security is one of the examples of the data level permissions implemented at database level. (<a href="https://www.oracle.com/uk/security/database-security/label-security/">https://www.oracle.com/uk/security/database-security/label-security/</a>).</p><blockquote>Roles and permissions protect data at the operational level.</blockquote><p>The next point to remember about is information sensitivity levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*GmWit_-Ib3N0VKweccw6hg.png" /></figure><p>On the diagram above you can see the information sensitivity levels: <strong><em>public</em></strong>, <strong><em>internal</em></strong>, <strong><em>restricted</em></strong> and <strong><em>secret</em></strong>. Public means the information is available for everyone without any restrictions and secret means no one else except a trusted group of people should have access to the given data. Definition of levels in between will vary between organisations depending on the requirements. The most important thing to note is the information can automatically flow (systems can request and process) from outside to the inside levels. Any attempt to request data from the more restricted levels to the less restricted is not permitted. There has to be a manual review process on the way of information out of the internal circle.</p><blockquote>Sensitivity levels protect data an organisation-wide level.</blockquote><p>Now, once we protected the data at operational and organisational levels, it’s time to talk about everyday routine tasks.<br>As a system architect one needs to establish the data access during the development and testing period. To make this possible, an architect needs to think about how to obfuscate information to have the volume of data similar to the production without any real information in it. It’s close to impossible to obfuscate data manually. At the same time organisation can not simply afford any real data leak.<br>Just an excerpt from the GPDR regulation:<br> “The less severe infringements could result in a fine of up to €10 million, or 2% of the firm’s worldwide annual revenue from the preceding financial year, whichever amount is higher.“ Source: <a href="https://gdpr.eu/fines/">https://gdpr.eu/fines/</a></p><blockquote>Obfuscator needs be a software written by developers at an organisation.</blockquote><p>The most important information to obfuscate is the information which can reference a real person or an entity. Names, addresses, emails, phone numbers, etc. must be changed to a randomly generated values.</p><p>Another operational risk is to have developers access the production environment for “for investigation”.</p><blockquote>No production access is allowed by any developer/architect at any time. Simply, there is no access.</blockquote><p>If one needs to get access and work with the real data he or she must follow the standard roles allocation procedures. No shortcuts must be allowed. Every access to production data must be logged (who, when, what) and regularly reviewed. Everyone who accesses production data must access it with his own id. No system or application accounts allowed for personal access.</p><p>One of the growing issues with the data is internal fraud. Organisation’s data is a valuable asset. Any leakage of the real data will result in fines for the organisation and reputation loss.<br>One way or another there will be people who have full access to the data. For example, database administrators have full access when they do database backup.</p><blockquote>The task the architect has is to minimize manual interventions and automate as much as possible.</blockquote><p>This will lower the risk of the theft of the information from an organisation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e48e34787f82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notes On Data — Quality]]></title>
            <link>https://medium.com/@sergmesh/notes-on-data-quality-1a08baaa813b?source=rss-10e05f301d16------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a08baaa813b</guid>
            <dc:creator><![CDATA[Sergey Meshkov]]></dc:creator>
            <pubDate>Tue, 02 Sep 2025 16:56:53 GMT</pubDate>
            <atom:updated>2025-09-02T16:56:53.119Z</atom:updated>
            <content:encoded><![CDATA[<h3>Notes On Data — Quality</h3><p><em>A series of remarks for a data architect to pay attention to when designing data management systems.</em></p><p>One ot the most challenging parameters to understand about data is its quality. What does it mean to have good quality data? Everyone wants good quality but how to define it on something non-tangible?<br>Let’s make a reference to a real product from our everyday life, a car for example. Everyone understands what a good quality car is:<br> — It fits for purpose of moving people around.<br> — It does not break often.<br> — If it breaks we have a way to fix it with genuine parts.<br> — It runs good.<br> — And so on.</p><p>More or less we understand what good quality means, even though it’s some kind of ephemeral parameter.</p><blockquote>Quality is a parameter with the value from the interval [0, 1] which indicates how much we trust the data. Value 1 means we fully trust it and 0 means no trust at all.</blockquote><p>There are multiple dimensions that can affect data quality. Let’s list some of them:<br><strong>Source</strong> — We get data from some kind of source to produce a new result from it. So, quality will include some information about the source and how much we trust the source itself. For this dimension we need to ask a question: do we know who provided source information for the required data to be calculated?<br>For example: If we get a name and address from an online survey, are we going to trust it much? On the other hand, if we get counterparty information from a trade made on exchange, what is going to be a trust level? As you can see the level of trust will be different as an exchange is a fully regulated environment.<br><strong>Time to live (ttl)</strong> — Has the data expired and how much time has left until it fully expires.<br><strong>Accuracy</strong> — This dimension checks how accurate the data is. Does the data represent the expected meaning? This is the dimension where validation and business rules might be applied to increase the trust level.<br>For example: Information typed in by a person for online survey has low trust. Let’s take “age” for the sake of explanation. It can not be greater than 120 years and less than 0. What if we will see age value bigger than 150 years? <br>One of the recent findings: “<a href="https://www.foxbusiness.com/politics/musk-claims-150-year-olds-receiving-social-security-benefits"><em>Billionaire Elon Musk on Tuesday claimed that a cursory review of Social Security records by the Department of Government Efficiency (DOGE) found evidence that the safety net program is paying benefits to 150-year-olds.</em></a>”<br><strong>Data format</strong> — Does the data fit the expected format?<br><strong>Completeness</strong> — Do we have all required fields populated to represent required object? <br>For example: “birthday” field should have the date, month and year fully populated to be complete.</p><p>The data quality parameter is defined as a number on purpose to remove any bindings to the nature of the data itself and make the quality value as abstract as possible.</p><p>The exact list of dimensions is specific to the nature of the data and locale a system manages.</p><p>Now let’s look at the data from a client perspective. Different clients might have different data quality expectations. Trading algorithms, for example, will require high quality data. While a surveys accumulation platform permits some inconsistencies. It all depends on the client’s requirements. And there has to be a service level agreement (SLA) between the data supplier and a client on the dimensions required and quality of data provided.<br>The practical approach is to have some basic dimensions applied by default (source, ttl, data format) with the rest defined by a client.</p><p>There is an international standard on the data quality called <strong>ISO 8000</strong> (<a href="https://en.wikipedia.org/wiki/ISO_8000">https://en.wikipedia.org/wiki/ISO_8000</a>).<br>It’s worth reading as it has additional insights on the process of data quality management.</p><p>From a system design perspective, any data storage solution has to have a module that validates existing data and assigns quality value to a data record on a regular basis. This indicates how good are the processes around the data management and how well established the system is to manage it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a08baaa813b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notes On Data — Time]]></title>
            <link>https://medium.com/@sergmesh/notes-on-data-time-ba27b0b0db07?source=rss-10e05f301d16------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba27b0b0db07</guid>
            <dc:creator><![CDATA[Sergey Meshkov]]></dc:creator>
            <pubDate>Sat, 30 Aug 2025 10:59:54 GMT</pubDate>
            <atom:updated>2025-08-30T10:59:54.188Z</atom:updated>
            <content:encoded><![CDATA[<h3>Notes On Data — Time</h3><p><em>A series of remarks for a data architect to pay attention to when designing data management systems.</em></p><p>One of the crucial parameters of any data stored in a system is time. A record in a table that was written one year ago might not be valid anymore. Time is closely related to the quality of the data a system holds.</p><p>If we don’t pay attention to the time dimension then difficult questions might be asked:<br>- If data is not up to date why do we need to store it in the first place?<br>- How can we ensure that the upstream system that uses our data will produce correct results if the source data is not up to date?<br>- And so on.</p><p>As a data architect, one needs to ensure that a system holds actual data and has a way to update it. They also need to keep track of when the data needs be updated in the future.</p><blockquote>The time dimension is an integral part of any data stored in any system.</blockquote><p>Unfortunately, having a field called “updated_on” with timestamp is not enough to make a decision on when we need to update it in the future.</p><p>It is possible to have time to live (ttl) configured at a system level or at a data level.</p><p>If we have “updated_on” at a data level and a system knows that it needs be updated after X amount of time, then we can answer the question on when we need to update it. This can be achieved by supplying a system-wide configuration parameter which tells how long the data is valid for. “updated_on” + system parameter defines when the data needs be updated.</p><p>If we have “ttl” at a data level then we need to have 2 columns “updated_on” and “ttl” which is enough to know when to update.</p><p>It is also possible to implement time to live by having two columns “valid_from” and “valid_to”. This clearly indicates validity period at a data level.</p><p>Another solution is to have single “expires_on” field that is being calculated at the data creation time and stays unchanged.</p><blockquote>As you can see from the example solutions above, a system has to have a module that monitors the data expiration and updates records on a regular basis. A data architect needs to keep in mind that such module should be a key part of any system.</blockquote><p>Some systems (depending on the requirements) might need to answer the question on: how did it happen that the upstream system has this result calculated?</p><p>Such requirements are related to the historical view of the data present in a system. So, you are not only supplying the actual data as shown above, but you also need to provide a way to see how the previous values were calculated. In this case the system should not overwrite the old data with the new data, and instead, it needs to offload it into a separate storage and make it available for historical queries. For such requirements validity periods become a key element of a storage solution design.<br>For example, table design like below guarantees that we have a way to answer “what was the price?” for any historical request`:<br> “product”, “price”, “valid_from”, “valid_to”</p><p>The time parameter could also be used to implement cleanup operations to shrink the data size a system holds in memory or on a disk to minimize load on the system and perform search operations faster. Multiple solutions are available to implement this, starting from time based data partitioning and finishing with storing expired data on a different physical server.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba27b0b0db07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notes On Data — Context]]></title>
            <link>https://medium.com/@sergmesh/notes-on-data-context-e76c27dfef0d?source=rss-10e05f301d16------2</link>
            <guid isPermaLink="false">https://medium.com/p/e76c27dfef0d</guid>
            <dc:creator><![CDATA[Sergey Meshkov]]></dc:creator>
            <pubDate>Thu, 28 Aug 2025 19:30:18 GMT</pubDate>
            <atom:updated>2025-08-28T19:30:18.087Z</atom:updated>
            <content:encoded><![CDATA[<h3>Notes On Data — Context</h3><p><em>A series of remarks for a data architect to pay attention to when designing data management systems.</em></p><p>Let’s imagine a dialog below:</p><p>— John, what’s the value ?<br> — 25.20<br> — 25.20 what ?<br> — what value ?</p><p>This dialogue is remarkable as it demonstrates the importance of context for any value we use. It does not matter where and how the 25.20 will be stored in a database, file or an object store. And storing it does not move us closer to understanding on what it actually is. We simply do not know why we need this number and how to use it. To solve the context issue we can create a database table and name a column as “exchange rate”. This will bring some context and by looking at just a column name we can perform checks on the data we store in the column.</p><p>Let’s say we created a table with 3 columns:</p><p><em>“from currency”, “to currency”, “exchange rate”</em></p><p>Just naming columns properly bring a lot of context into the picture, and we can start working with the data in this table.</p><p>On the other hand, if we name the “exchange rate” column as “room area”:</p><p><em>“from currency”, “to currency”, “room area”</em></p><p>The context is completely ruined. It is an extreme example but anyone looking at this table will not be able to figure out why currencies have room area. Sometimes “small” correction to naming can destroy context in a big project and cause a lot of misunderstanding.</p><p>This brings an important requirements to the context definition.</p><blockquote><strong>Context has to be consistent and clearly defined.</strong></blockquote><p>There re multiple ways to define context to data, i.e. schema, table structure, etc. Context (or metadata) has to be present and be available along the data.</p><p>An example of clearly defined context: <br>25.20 is an exchange rate of pound sterling to mexican peso taken on the 28th of August 2025 at 18:40 BST. <br>Source: <a href="https://share.google/2g0CCnuImOUoycHRI">https://share.google/2g0CCnuImOUoycHRI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e76c27dfef0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>