<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kyle Martin on Medium]]></title>
        <description><![CDATA[Stories by Kyle Martin on Medium]]></description>
        <link>https://medium.com/@kyle_martin?source=rss-9ed8c77fd5aa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*eKGFB5sMKEMN-5WcM_JalQ.png</url>
            <title>Stories by Kyle Martin on Medium</title>
            <link>https://medium.com/@kyle_martin?source=rss-9ed8c77fd5aa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:01:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kyle_martin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MongoDB in production: How connection pool size can bottleneck application scale]]></title>
            <link>https://medium.com/@kyle_martin/mongodb-in-production-how-connection-pool-size-can-bottleneck-application-scale-439c6e5a8424?source=rss-9ed8c77fd5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/439c6e5a8424</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[mongo]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Kyle Martin]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 01:41:27 GMT</pubDate>
            <atom:updated>2020-01-07T01:41:27.750Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BFiVxZgItkcV6NbghtPgA.jpeg" /></figure><blockquote>Understanding how MongoDB connection pools and pool sizing works is a fundamental part of running an effective MongoDB client application at global scale.</blockquote><blockquote>This article looks at what connection pool size is in MongoDB and how we can test the effect the pool size setting has on the time taken to complete queries, where if set incorrectly, can lead to hidden application bottlenecks by complex queries slowing down simple ones.</blockquote><p>As your application needs to scale, the number of queries to MongoDB increases. The complexity of your application also grows, so query requirements become larger — so do potentially hidden bottlenecks.</p><p>Let’s say you’re working on an application with two types of users:</p><p>1 ) General users of the application, who expect fast loading times. These users account for the majority of application traffic.</p><p>2 ) Administrators of the system who need to perform tasks which can span across thousands of users and their related items — administration actions which are usage intensive and resource heavy.</p><p>The first user set(general users) are paying customers, of which there are thousands at any point in time. Any actions these users do in the application needs to be extremely fast — such as a logging in, viewing pages, and other application features. The database queries required for these features are very small, and should be very fast.</p><p>The application features used by the second user set(administrators) will generally be very working with large amounts of data, which in turn requires large and complex database queries.</p><p>After reports from users that the application has become slower, you start looking at application metrics. You find that MongoDB queries have suddenly become very slow — even for indexed findOne queries which use to be extremely fast.</p><p>To work out what’s happened here, we need to understand MongoDB pool sizing.</p><p><strong>Example:</strong></p><p>A full sample application below can be<a href="https://github.com/js-kyle/mongodb-pooling"> found on Github.</a><em> </em>This article’s code samples focus on the Node.js MongoDB driver, however the underlying concepts remain the same across the various MongoDB drivers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74311fa7a448a69732414b06c5d4f8d7/href">https://medium.com/media/74311fa7a448a69732414b06c5d4f8d7/href</a></iframe><p>In the above code sample, we have a minimal <a href="https://expressjs.com/">Express.js </a>server with two endpoints, /slow and /fast . One database connection is created, which is then shared for all MongoDB queries — this is much more performant than opening and closing TCP connections for each and every query the application makes to the database.</p><p>As we are now sharing the database connection across the application, we need to understand how the pool size option of this connection comes into play. The pool size is the option which determines how many queries may be run using the same connection in parallel, similar to the number of lanes in a tunnel. An incorrectly set value for this connection can bottleneck the whole application.</p><p>The /slow route uses an example query which will take 5 seconds to return a result — this is designed to emulate a slow query in your application such as an administrator loading every user, or a todo list application loading a large number of items.</p><p>The /fast route is an example query that should return within a small number of milliseconds — this is designed to emulate a very simple, yet critical user facing query in your application, for example, a general user logging in.</p><p>In the code sample, the environment variable MONGO_POOLSIZE can be used to dynamically change the pool size on application start. With this, we can begin to benchmark our application with various pool sizes configurations. To accurately test this, we can use the application load testing tool <a href="https://artillery.io/">artillery.io</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e58f637287ee7d1d6b0741bef310d872/href">https://medium.com/media/e58f637287ee7d1d6b0741bef310d872/href</a></iframe><p>The above yaml configuration file can be used with the artillery.io cli to run a performance test. This performance test will target our running application at http://localhost:3000 and emulate a new user arriving at the application every second, for 5 seconds. Each user will visit both the /slow and /fast endpoint and wait for a response from the server.</p><p>First, let’s run the loadtest using a pool size of 3. This means that our database connection will be able to run a maximum of three queries to MongoDB at one time.</p><p>$ MONGO_POOLSIZE=3 node index (Starts the application, with a pool size of 3)</p><p>$ artillery run loadtest.yaml</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/1*LylRdmi3JQJyoS2OrKBTOg.png" /></figure><p>With the pool size set to three, all of the tests completed successfully, with a maximum response time of 9017ms (our slow query emulates one which takes 5000ms).</p><p>What happens when we lower our pool size to 1? This means that our database connection will only be able to run one MongoDB query at one time.</p><p>$ MONGO_POOLSIZE=1 node index</p><p>$ artillery run loadtest.yaml</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*IkgHAECWeqAm-A8OVo_doA.png" /></figure><p>Though every request succeeded, the maximum response time increased from 9017ms to 21142.7ms. Why? <strong>Our concurrent requests were now blocked by the first slow query, so all of our fast queries have suffered as a result.</strong></p><p>What happens if we increase the pool size to 10? This means that our database connection will be able to run up to ten MongoDB queries at one time.</p><p>$ MONGO_POOLSIZE=10 node index</p><p>$ artillery run loadtest.yaml</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*0bZ7W7LWg4Xx_Jk5gRI9zA.png" /></figure><p>Our maximum response time is now 5019ms — the time required to run our example slow query. This shows that no other queries were blocking the database connection during this test, meaning that our /slow and /fast endpoints provided the expected result to the user without being bottlenecked.</p><p>So, what is this pool size setting, and why is it stalling my application scale?</p><p>Picture the MongoDB connection as a tunnel. The pool size determines how many lanes of traffic can enter the tunnel at a time. If you have a one-lane tunnel with a bus and a Ferrari, the Ferrari can’t get through the tunnel until the bus has. This means that sometimes, a slow bus can make even a Ferrari slow right down if there aren’t enough lanes.</p><p>When we add extra lanes, i.e. increasing our connection pool size, we can make room for the Ferrari to speed through and not get held up by the slow bus.</p><p><strong>What should I do about it?</strong></p><p>If you expect your application to require long running queries, or many concurrent queries you should benchmark your application by simulating expected production load with a tool such a <a href="https://jmeter.apache.org/">JMeter</a> or<a href="https://artillery.io/"> artillery.io</a> to investigate the impact of increasing the pool size for the MongoDB connection. The optimal pool size for your application will also need to take into account other clients which are sharing the same MongoDB server — as resource usage on the MongoDB server should also be monitored.</p><p>Tools such as Azure Application Insights can be used to instrument the MongoDB query performance in your application as detailed in <a href="https://medium.com/@yoad/auto-mongodb-telemetry-in-application-insights-too-verbose-quick-note-fb4af3470ff0">this blog post</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=439c6e5a8424" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Load Testing Socket.IO Web Applications and Infrastructure]]></title>
            <link>https://medium.com/@kyle_martin/load-testing-socket-io-web-applications-and-infrastructure-3e96011898e0?source=rss-9ed8c77fd5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e96011898e0</guid>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[socketio]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Kyle Martin]]></dc:creator>
            <pubDate>Sun, 08 Sep 2019 03:59:12 GMT</pubDate>
            <atom:updated>2019-09-30T00:23:30.799Z</atom:updated>
            <content:encoded><![CDATA[<h3>Load Testing socket.io Web Applications and Infrastructure</h3><h4>Tutorial for load testing your socket.io WebSockets server</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aRlozFzadEGW-xRHsACY0w.jpeg" /></figure><p>Are you shipping a scalable real-time back end? This article provides a tutorial for load testing your <a href="https://socket.io/">socket.io</a> based WebSocket server, to simulate production load and identify performance bottlenecks or scalability issues<em>.</em></p><p><a href="https://socket.io/">socket.io</a> is a framework built on top of the WebSocket protocol, which upgrades an HTTP connection using an Upgrade: websocket header, initiating a WebSocket handshake.</p><p>This provides a persistent connection between a client and server where both parties can emit messages at any time, without the need for the client to poll the server.</p><p>socket.io adds additional functionality on top of this, such as increasing browser and device support of your application by providing an option to fall back to Ajax requests when using a client which does not support WebSockets, or a client on a network which restricts the upgrade HTTP header or protocol changes.</p><h3>Setup</h3><p>To follow along with this tutorial, you can either clone and <a href="https://github.com/js-kyle/socketio-loadtesting">run the demo repository </a>locally (suggested), or you can use the <a href="https://socketio-loadtest.herokuapp.com/">version hosted by me on Heroku</a>.</p><p>If you choose to run the app locally, you will need to clone the <a href="https://github.com/js-kyle/socketio-loadtesting">repository</a> and install the required dependencies using npm. The npm start script will start the app on your configured PORT, or 8080 by default.</p><p>First, let’s get familiar with the application. Once running, the application should work as below.</p><p>The main features to note are being able to send a realtime message between users, and see a count of the current participants in the chat room.</p><p>Below is an example of two different users sending and receiving messages in realtime — great!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FeWMql6QYKwzJeLz62JgJg.gif" /><figcaption>Application demo</figcaption></figure><h3>Load Testing</h3><p>Now that we’ve set up the application, we can start our load testing. For this, we’ll use the <a href="https://artillery.io/">Artillery Community Edition</a> tool, which is a free and open-source load testing and functional testing toolkit.</p><p>To install the Artillery.io toolkit globally:</p><pre>$ npm install artillery -g</pre><p>Next, we need to create a <a href="https://yaml.org/">YAML</a> configuration file, which will specify the test configuration and scenarios.</p><p><a href="https://github.com/js-kyle/socketio-loadtesting/blob/master/load-test/simple-test.yaml">simple-test.yaml</a></p><pre>config:<br>  target: &quot;wss://socketio-loadtest.herokuapp.com&quot;<br>  socketio:<br>    transports: [&quot;websocket&quot;]<br>  phases:<br>    - duration: 10 # Run scenario for 10 seconds<br>      arrivalCount: 20 # Create 20 virtual users per scenario<br>scenarios:<br>  - engine: &quot;socketio&quot;<br>    flow:<br>      - emit:<br>          channel: &quot;add user&quot;<br>          data: &quot;John Doe&quot;<br>      - emit:<br>          channel: &quot;new message&quot;<br>          data: &quot;Hello! {{ $randomString() }}&quot;<br>      - think: 5 # do nothing for 5 seconds, then disconnect</pre><p>In the above configuration file, we specify our target URL, and Socket.IO engine. We configure our phases section to specify the number of virtual users we will emulate and the duration of the test. Initially, these are low numbers.</p><p>In the scenarios section, we specify what WebSocket events we would like the artillery.io client to emit to the server.</p><p>For this simple example, we will add a new user to the chat, using the <a href="https://github.com/js-kyle/socketio-loadtesting/blob/master/server.js#L33">add user</a> event in the demo application, then post a <a href="https://github.com/js-kyle/socketio-loadtesting/blob/master/server.js#L24">new message</a>. After five seconds, the user leaves which disconnects the WebSocket.</p><p>To run the above script, we will use the artillery run command.</p><pre>$ artillery run load-test/simple-test.yaml</pre><p>If we view our application in a browser (either locally, or at <a href="https://socketio-loadtest.herokuapp.com/">https://socketio-loadtest.herokuapp.com</a>), virtual users can be seen joining the chatroom and leaving messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7O5Q3Esa9iS-zs8BxKukSQ.gif" /><figcaption>Simple socket.IO load test</figcaption></figure><p>The above simple script can be modified as you wish to increase the number of users in a scenario by tweaking the configuration values. For a more advanced test, you can find more verbose scenario examples below.</p><h3>Assertions</h3><p>Now that we have the tool setup, we can set the task to fail based on certain conditions, which is great for a CI environment.</p><p>We can choose to set a maximum allowed latency in milliseconds for min, max, median, p95, and p99, or a maximum failure percentage rate. These can be configured to test again certain business rules, or SLOs.</p><p>To configure the above, we add an ensure entry to our configuration. Further examples can be found in the <a href="https://artillery.io/docs/script-reference/">artillery.io documentation</a>.</p><pre>config:<br>  target: &quot;ws://localhost:8080&quot;<br>  ensure:<br>    maxErrorRate: 1 # fail if error rate exceeds 1%<br>    max: 500 # fail if max response time exceeds 500ms</pre><h3>Advanced Configuration</h3><p>In the advanced example, dynamic variables are used to simulate a scenario which is closer to how the application will be used in production (real data). This is done using the <a href="https://www.npmjs.com/package/faker">faker</a> npm module, inside a <a href="https://github.com/js-kyle/socketio-loadtesting/blob/master/load-test/custom.js">custom processor</a> which allows us to run our custom code.</p><p>The load phase is configured to increase the arrival rate from 10 to 50 users over two minutes, followed by 10 minutes at 50 new users per second.</p><p><a href="https://github.com/js-kyle/socketio-loadtesting/blob/master/load-test/advanced-test.yaml">advanced-test.yaml</a></p><pre>config:<br>  target: &quot;ws://localhost:8080&quot;<br>  ensure:<br>    max: 500 # fail if max response time exceeds 500ms<br>    maxErrorRate: 1 # fail if error rate exceeds 1%<br>  socketio:<br>    transports: [&quot;websocket&quot;]<br>  processor: &quot;./custom.js&quot; # set a processor for dynamic variables<br>  phases:<br>    - duration: 120<br>      arrivalRate: 10<br>      rampTo: 50<br>      name: &quot;Warm up phase&quot;<br>    - duration: 600<br>      arrivalRate: 50<br>      name: &quot;Sustained max load&quot;<br>scenarios:<br>  - engine: &quot;socketio&quot;<br>    flow:<br>      - function: &quot;getChatData&quot; # load variables<br>      - emit:<br>          channel: &quot;add user&quot;<br>          data: &quot;{{ name }}&quot;<br>      - emit:<br>          channel: &quot;new message&quot;<br>          data: &quot;{{ greeting }}&quot;<br>      - think: 10 # stay connected for 10 seconds<br>      - emit:<br>         channel: &quot;new message&quot;<br>         data: &quot;{{ goodbye }}&quot;</pre><p>Run artillery using the advanced-test.yaml file</p><pre>$ artillery run load-test/advanced-test.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xS97tVPZAf7c50NQ1BQNAA.gif" /><figcaption>Advanced socket.IO load test</figcaption></figure><p>The above is a snippet of the ramp-up phase defined in the configuration file, where the <em>arrival rate</em> of virtual users increases from 10 to 50 users over two minutes.</p><p>The advanced scenario example can be tweaked in many ways depending on what type of test you want to run, such as how many concurrent connections can be handled, or maximum number of users before performance begins to degrade.</p><h3>Recap</h3><p>Load testing our Socket.IO applications is an important step in the application lifecycle, to learn:</p><ul><li>The number of concurrent WebSocket connections the application can support.</li><li>How the application handles failure.</li><li>Determines if the current infrastructure is sufficient for the expected load.</li><li>Gives confidence in the system and its reliability and performance.</li><li>Helps ensure that degradation of any component in the stack doesn’t lower the state of security.</li></ul><p>Next in the Socket.IO blog series will be a tutorial on how to debug memory leaks in Socket.IO applications. If you have any ideas on similar topics you’d be interested in, feel free to get in touch with those ideas.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e96011898e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding and protecting against malicious npm package lifecycle scripts]]></title>
            <link>https://medium.com/@kyle_martin/understanding-and-protecting-against-malicious-npm-package-lifecycle-scripts-8b6129619d7c?source=rss-9ed8c77fd5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b6129619d7c</guid>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[yarn]]></category>
            <dc:creator><![CDATA[Kyle Martin]]></dc:creator>
            <pubDate>Tue, 27 Aug 2019 08:35:06 GMT</pubDate>
            <atom:updated>2019-08-29T21:14:22.107Z</atom:updated>
            <content:encoded><![CDATA[<p>This article explains the importance of <em>understanding</em> how npm lifecycle scripts such as preinstall and postinstall work, how they could, (and have been) used maliciously, and offers some advice on what you can do about it as part of your npm project health.</p><h3>Issue</h3><p>If you are using <a href="https://npmjs.com/">npm,</a> the popular JavaScript package manager, you have likely heard about the <a href="https://www.npmjs.com/package/eslint-scope">eslint-scope</a> package attack, where control over the heavily relied upon package was obtained by a malicious actor, who published a new version containing malicious code. That malicious code was executed using a postinstall lifecycle script, meaning each user who installed the package potentially had their npm registry login details sent to a remote address by the malicious script. The official npm incident report can be found <a href="https://status.npmjs.org/incidents/dn7c1fgrr7ng">here</a>, but the important detail is that unknowingly, users running the familiar npm install command had no idea that this malicious code was being executed during the postinstall step. As these scripts run by default, any background tasks they execute are often hidden from the user, who is only looking for feedback on whether or not the new thing they installed is ready.</p><p><strong>Example:</strong></p><pre>// package.json file</pre><pre>&quot;name: &quot;express-example-package&quot;,<br>&quot;scripts&quot;: {<br>  &quot;postinstall&quot;: &quot;node malicious.js&quot;<br>}</pre><p>If the above package.json file was an example package on the npm registry, any user who decided to npm install express-example-packagewould have the above postinstall script executed - the contents of this script file could be anything! The same behaviour is true for the scripts preinstall , postinstall , preuninstall and postuninstall. The npm reference for these scripts are available <a href="https://docs.npmjs.com/misc/scripts">here</a>.</p><h3>Prevention</h3><p>In reality, preventing this issue is much harder than one simple solution. These lifecycle hooks are an important feature - they can help set up packages in complex ways and perform important cleanup or preparation tasks, so it can be limiting to simply opt out of running these hooks. Due to this, the recommended approach to prevent this issue will always be to <strong>1)</strong> <strong>review dependencies carefully, and use a lockfile to prevent auto-installing new packages</strong></p><p><strong>2) Opt out of running scripts on install</strong></p><p>When installing a package, you can chose to opt out of running scripts using the ignore-scripts option in npm or Yarn:</p><p>npm install --ignore-scripts</p><p>yarn add --ignore-scripts</p><p>Or, if you never want to run these scripts when installing packages, you can modify your global npm configuration:</p><p>npm config set ignore-scripts true</p><p>yarn config set ignore-scripts true</p><p>Note: this setting could also be added to a project’s .npmrc or .yarnrc file.</p><p><strong>3) Audit current modules script use</strong></p><p>I found a lack of tooling available to see which dependencies in the current dependency tree are currently running lifecycle scripts, so have published a cli tool to view this information. <a href="https://www.npmjs.com/package/npm-viewscripts">It is available on npm.</a></p><p>To use the tool, you can install it globally npm install npm-viewscripts -g .</p><p>Inside a project directory, running npm-viewscripts will provide a list of any dependencies currently installed which have, and will continue to run a script on install or uninstall.</p><h3>My thoughts</h3><p>This type of vulnerability is not a fault of npm, as these lifecycle scripts are a very helpful feature for package management, however the risks need to be understood by users. It could be argued that the issue has a wider scope with npm due to the large size of a typical Node.js application’s dependency tree, and the philosophy of building and sharing and consuming many small and discrete modules.</p><p>I think it is important that the community, and in particular maintainers of popular packages take precautions to prevent these attacks from continuing - the best prevention against this is to enable 2FA for all npm accounts with write access to packages. With this enabled, any npm credentials which are stolen by one means or another have less harm on consumers of the package, but that of course does nothing to prevent a trusted maintainer suddenly going rogue.</p><p>npm are doing some great work to reduce the risk of these types of attacks by reducing typosquatting, and offering free security tooling such as npm audit .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b6129619d7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>