Table of Content

1. What Are Core Web Vitals
2. Why Core Web Vitals Matter
3. Key Metrics of Core Web Vitals
   3.1 Largest Contentful Paint (LCP)
   3.2 Cumulative Layout Shift (CLS)
   3.3 Interaction to Next Paint (INP)
   3.4 CWV Scoring and Penalties
   3.5 Other Important Metrics
4. Enhancing & Testing CWV
   4.1 How to Debug and Test CWV
   4.2 Mobile-First Indexing Context
   4.3 Accessibility and CWV
5. Real-World Performance Monitoring
   5.1 Balancing Lab Data and Field Data
   5.2 Geographic Considerations in Field Data
   5.3 Monitoring CWV
   5.4 How we measured CWV at The Telegraph
6. Troubleshooting Common Issues and Pitfalls
7. FAQs About CWV
8. Useful Resources

1. What Are Core Web Vitals?

Core Web Vitals (CWV) are performance metrics introduced by Google to measure user experience on websites. These metrics assess aspects like loading speed, interactivity, and visual stability. Optimizing these metrics is critical because they directly influence Google’s ranking algorithm as part of the “Page Experience” signals introduced in 2021.

2. Why do Core Web Vitals Matter?

1. Enhanced User Experience: Fast, responsive, and visually stable websites create better user experiences, increasing engagement and conversions.
2. Improved SEO Rankings: CWV is a ranking factor for Google. A site with poor scores risks lower search engine visibility.
3. Reduced Bounce Rates: Pages that meet CWV targets minimise user frustration, leading to fewer users returning to search results.
4. Future-Proofing: As CWV evolves alongside web technology, maintaining good scores ensures your site remains competitive.

3. Key Metrics of Core Web Vitals

The three main CWV metrics are:

3.1 Largest Contentful Paint (LCP):

• Definition: Measures the render time of the largest visible content element in the viewport.
• Target: Less than 2.5 seconds (PageSpeed Insights).
• Importance: LCP indicates how quickly a page renders its most prominent visual element, which typically includes large images, videos, or block-level text. While this element is often central to the page’s purpose, it may not always represent the primary content from a user experience standpoint.
• Common Issues: Large images, slow server response, render-blocking JavaScript or CSS.
• How to Improve: Optimize and preload critical resources to ensure they load as early as possible, reducing delays in rendering key content. Compress images and serve them in modern formats like WebP or AVIF to improve load times, while using Content Delivery Networks (CDNs) helps minimize latency by delivering assets from servers closer to users. Also, reducing the main thread workload by deferring non-critical scripts and lazy-loading offscreen images can significantly enhance performance. Minifying and optimizing CSS and JavaScript further prevents render-blocking issues, ensuring a smoother and faster user experience.

3.2. Cumulative Layout Shift (CLS):

• Definition: CLS measures layout stability by quantifying unexpected layout shifts that occur during a page’s lifespan.
• Target: A score below 0.1 (PageSpeed Insights).
• Importance: Prevents content from moving unexpectedly, which can disrupt user interaction.
• Common Issues: unoptimized animations, dynamic ads, or images without defined dimensions.
• How to Improve: Define width and height for all media elements to prevent unexpected shifts, and use CSS aspect-ratio to maintain consistent dimensions for images and videos. Reserving space for ads and dynamic content ensures they do not push other elements when loading. Additionally, using font-display: swap helps avoid invisible text while web fonts load, reducing visual instability.

3.3. Interaction to Next Paint (INP):

• Definition: This measure measures responsiveness by evaluating the delay between user interactions and page updates, including the time it takes for visible response or feedback to appear.
• Target: Less than 200 milliseconds (PageSpeed Insights).
• Importance: Ensures users can interact with the page smoothly without noticeable delays.
• Common Issues: JavaScript delays, heavy scripts, and large files blocking interaction.
• How to Improve: Break up long tasks using code-splitting techniques; optimize JavaScript and defer non-critical scripts; use modern frameworks prioritising responsiveness.

3.4 CWV Scoring and Penalties

Each metric contributes to an overall Core Web Vitals score, which is out of 100. Google evaluates these scores to determine how well a site meets user experience standards:

• Good Overall Score (90–100): Indicates excellent performance and compliance with Google’s expectations. No penalties apply.
• Needs Improvement (50–89): Scores in this range suggest areas for optimization. While not directly penalized, poor user experience can indirectly impact rankings.
• Poor Score (<50): Sites with overall scores below 50 risk reduced visibility in search engine results due to failing to meet user expectations.

Key Penalty Thresholds:

• Individual Metrics: If any of the three core metrics (LCP, CLS, or INP) consistently fall into the “Poor” range, Google may penalize the site with lower rankings.
• Overall Score: Maintaining an overall CWV score below 75 can hinder competitiveness and long-term SEO performance.

3.5 Other Important Metrics

In addition to CWV, several supplementary metrics help provide a comprehensive performance overview:

• Time to First Byte (TTFB): Measures server response time.
• Time to Interactive (TTI): Time until the page becomes fully interactive.
• Speed Index (SI): Shows how quickly the content is visually displayed during load.

4. Enhancing and Testing CVW

4.1 How to Debug and Test CWV

Tools for Testing

1. Google Lighthouse:

• A lab testing tool available in Chrome DevTools. Run it to get actionable recommendations for improving performance.

“The Chrome DevTools Lighthouse panel.”

2. PageSpeed Insights:

• Combines field and lab data to offer a comprehensive view of page performance.
• Leverages the Chrome User Experience Report (CrUX) for real-world user metrics.

Left: “PageSpeed Insights panel example”; Right: “CrUX Dashboard Panel example”

3. Chrome DevTools Performance Tab:

• Simulates different network conditions and devices for detailed analysis.

Chrome DevTools Performance Tab

4. Third-Party Tools:

• Web Vitals Chrome Extension: Monitors CWV metrics in real-time while browsing.
• SpeedCurve: Offers advanced tracking and performance insights.
• DataDog: Tracks performance and monitors CWV trends.
• Requestly: Useful for debugging and simulating network conditions.
• WebPageTest– A powerful tool for in-depth performance analysis, waterfall breakdowns, and testing across different network conditions and locations.

Simulating Different Environments

• Test under various network conditions (Wi-Fi, 4G, 3G).
• Assess performance across different devices:
• Desktop: Windows, macOS.
• Mobile: Android, iOS.

4.2 Mobile-First Indexing Context

Google evaluates sites primarily based on their mobile performance. If a site performs poorly on mobile, it can negatively affect rankings, even if the desktop experience is excellent.

Key Mobile-Specific Strategies:

1. Optimize for Smaller Screens:

• Ensure images are appropriately sized for mobile screens using responsive design techniques like srcset.
• Test font sizes to maintain readability without zooming.

2. Test Under Realistic Mobile Conditions:

• Simulate slow network speeds (e.g., 3G or 4G) using tools like Chrome DevTools to identify bottlenecks.
• Use Lighthouse in mobile emulation mode to analyze performance.

3. Minimize Mobile-Specific Issues:

• Avoid intrusive interstitials that can disrupt the mobile user experience.
• Prioritize above-the-fold content to load first for mobile users.

Tools for Mobile-First Indexing:

• Google Mobile-Friendly Test: Quick check for mobile usability issues.
• WebPageTest: Allows detailed testing under mobile conditions with different network speeds.

4.3 Accessibility and CWV

How CWV Enhancements Benefit Accessibility:

1. Improved Load Times:

• Faster loading pages reduces frustration for users relying on assistive technologies like screen readers. These devices can perform more efficiently when pages load quickly and provide structured content.

2. Visual Stability (CLS):

• Predictable layouts help users navigate easily, especially those with cognitive disabilities. Sudden layout shifts can disorient users relying on visual or keyboard navigation aids.

3. Interactive Performance (INP):

• Delays in interactivity impact users relying on screen readers or keyboard-only navigation. Ensuring responsive interactions improves the usability of web elements like forms, buttons, and menus.

Actionable Accessibility Improvements:

• Use semantic HTML elements (e.g., <header>, <nav>, <main>) for better screen reader support.
• Define clear focus states for interactive elements to help keyboard users.
• Optimize ARIA (Accessible Rich Internet Applications) attributes alongside CWV improvements to ensure compatibility with assistive technologies.
• Test accessibility using tools like:
• WAVE Accessibility Tool — identifies accessibility issues.
• Axe DevTools — integrates into browser dev tools for accessibility auditing.

5. Real-World Performance Monitoring

Understanding website performance requires combining lab data and field data for a comprehensive view of strengths and weaknesses under various conditions.

5.1 Balancing Lab Data and Field Data

Lab Data: Tools like Google Lighthouse and Chrome DevTools test performance in controlled environments (e.g., high-speed networks, specific devices). They identify technical bottlenecks but don’t fully represent real-world user experiences.

Field Data: Real User Monitoring (RUM) tools like CrUX and SpeedCurve collect metrics from actual users, reflecting performance across diverse conditions, including geographic location, device type, and network speed.

Key Differences

• Environment: Lab data uses ideal conditions; field data accounts for real-world factors like geography and internet speed.
• Granularity: Lab data provides specific recommendations; field data shows broader trends.
• Purpose: Lab data aids proactive optimization; field data validates improvements and monitors trends.

5.2 Geographic Considerations in Field Data

Performance varies by user location due to server distance and network infrastructure. Latency increases with distance, especially in regions with weaker networks. Field data helps identify slow regions, enabling localized optimizations like CDNs or edge servers to reduce latency.

5.3 Monitoring CWV

1. Field Monitoring:

• Use tools like Chrome User Experience Report (CrUX) and SpeedCurve to gather real-world performance data.

2. Set performance budgets:

• Define acceptable thresholds for metrics. Use alerts in tools like DataDog to monitor deviations.

3. Integrate in CI/CD Pipelines:

• Automate performance tests to ensure new deployments don’t negatively impact CWV.

5.4 How we measured CWV at The Telegraph

At The Telegraph Media Group (TMG), monitoring and improving CWV is essential for ensuring an optimal reading experience — especially on content-heavy pages where layout stability, load time, and responsiveness matter deeply.

To effectively measure and track CWV metrics like LCP, CLS, and INP, TMG employs a combination of synthetic and real-user monitoring tools, including:

• Google Chrome User Experience Report (CrUX)
• PageSpeed Insights
• WebPageTest
• Chrome DevTools
• Datadog (for alerting and anomaly detection)

This hybrid approach helps not only catch regressions before they reach production but also understand performance across various devices, templates, and geographic locations.

By tracking and valiadating improvements over time by integrating performance monitoring directly into development workflows and using internal dashboards for visibility.

For a full breakdown of how The Telegraph approaches CWV optimization — including how they resolved layout shifts— check out the following full case study on web.dev.

6. Troubleshooting Common Issues and Pitfalls

Optimizing CWV requires understanding common challenges, avoiding frequent mistakes, and taking actionable steps. Here’s a consolidated guide:

1. For CLS Drops:

• Ensure all images and videos have defined dimensions.
• Use CSS animations instead of JavaScript for smoother transitions.
• Test with slow networks to spot problematic layout shifts.

2. For LCP Drops:

• Optimize server response times.
• Compress and preload critical assets (images, fonts).
• Minify CSS and JavaScript files.

3. For INP Drops:

• Reduce JavaScript execution time by lazy-loading scripts and splitting large bundles.
• Defer non-critical JavaScript to improve INP.
• Use efficient event handlers to minimize delays.
• Monitor long tasks in the Performance tab of Chrome DevTools.

4. General Optimization

• Minify CSS, JavaScript, and HTML files.
• Use a Content Delivery Network (CDN) to reduce server response times.
• Enable browser caching for reusable assets.

5. Common Pitfalls to Avoid:

• Overoptimizing for a single metric without considering holistic user experience.
• Neglecting mobile testing in favour of desktop.
• Failing to account for the impact of third-party scripts or heavy assets.

7. FAQs About CWV

1. What happens if my site fails the CWV assessment?

• Poor CWV scores can negatively impact your search rankings and user retention.

2. Can CWV be tested in staging environments?

• Yes, using lab tools like Lighthouse or WebPageTest.

3. How often should CWV be monitored?

• Continuously monitor using tools like SpeedCurve and DataDog and set performance budgets for ongoing checks.

4. How do third-party scripts affect CWV?

• Third-party scripts can significantly slow down performance. Use a tag manager to control them and load only necessary scripts.

8. Useful Resources

• Google’s Core Web Vitals Documentation
• PageSpeed Insights
• PageSpeed Insights Docs
• Lighthouse Performance Overview
• CrUX Dashboard

By understanding and optimizing Core Web Vitals, you ensure that your website meets Google’s standards and provides a superior user experience. This is crucial for maintaining competitive SEO rankings and driving long-term engagement.

Understanding Core Web Vitals (CWV): Why They Matter for Your Website’s SEO was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post breaks down these three essential types of testing, exploring their descriptions, tools, advantages, disadvantages, and decision-making criteria for choosing the right mix for your project. Let’s demystify these testing layers and understand how they contribute to software quality.

Testing Layers Simplified: Speed, Scope, Coverage, and Cost

Testing is often represented as a pyramid, as shown in the above figure, where the foundation consists of unit tests, followed by integration tests, and topped with E2E tests. This visual helps balance testing speed, scope, coverage — and cost. The cost of identifying and fixing bugs increases exponentially as you move up the testing pyramid.

• Unit Tests: Fast, inexpensive, and numerous, targeting individual components. Bugs caught here are the easiest and cheapest to fix because the scope is limited, dependencies are minimal, and debugging is straightforward.
• Integration Tests: Moderate in speed and cost, focusing on the interactions between components. Bugs identified here cost more to fix than in unit tests due to increased complexity and dependencies.
• E2E Tests: Slow and costly to execute, replicating real-world workflows. Bugs discovered at this level are the most expensive to fix as they require debugging through multiple layers of the system and possibly redesigning workflows.

Striking the right balance between these layers ensures efficient testing strategies. By addressing bugs as early as possible, particularly during unit testing, development teams can save significant time and money.

Unit Testing: The Building Blocks of Software Quality

What is Unit Testing?

Unit testing focuses on individual components of code, such as functions or methods, testing them in isolation from the rest of the system. It ensures that the smallest building blocks of an application behave as intended.

Tools and Frameworks

Popular tools for unit testing include:

• JUnit: A widely used framework for Java applications.
• NUnit: A robust testing framework for .NET applications.
• Pytest: A flexible and powerful testing tool for Python.
• Jest: A popular framework for JavaScript and TypeScript, especially suited for React, Vue, and Node.js.

Use Cases

Unit tests are ideal for verifying the following:

• Single functions or methods.
• Classes and their associated behaviours.
• Core algorithms or logic in the system.

Advantages

• Speed and Cost-Effectiveness: Unit tests execute quickly, providing immediate feedback. Early detection of bugs at this level is far cheaper compared to addressing issues discovered later in integration or E2E testing.
• Simplicity: Easy to write and maintain, enabling developers to efficiently cover large portions of the codebase.

Disadvantages

• Limited Scope: They focus on isolated components and may miss issues related to interactions between system parts.

Example

Testing a function that calculates discounts for a shopping cart or validating an algorithm that filters search results.

Challenges and Mitigation

Dependency issues can arise when testing complex components. Mocking or stubbing external dependencies can resolve these problems.

Connection to TDD

Unit testing is integral to Test-Driven Development (TDD), a methodology where tests are written before the actual code. TDD ensures code correctness and encourages developers to design testable, modular code. In TDD:

1. Developers write a failing test that defines a new function or improvement.
2. They write just enough code to pass the test.
3. The code is refactored while ensuring the test still passes.

This iterative process ensures that all code has corresponding tests, promoting better design, early bug detection, and robust software.

Integration Testing: Connecting the Dots

What is Integration Testing?

Integration testing examines how different components or modules of an application work together. It uncovers issues such as data mismatches, interface incompatibilities, and misconfigured dependencies.

Tools and Frameworks

Effective tools for integration testing include:

• TestNG: Known for its flexibility and robust reporting capabilities.
• Spock: A powerful testing and specification framework for Java and Groovy.
• Pytest: Extensible enough to handle integration testing.
• Jest: Frequently used for testing interactions between components in JavaScript applications.

Use Cases

Integration tests are commonly applied to scenarios like:

• Communication between microservices or distributed systems.
• Database interactions to validate data consistency and correctness.
• Verifying API integration with other components.

Advantages

• Broader Coverage: Catches issues that unit tests might overlook, such as problems with data flow or interface mismatches.
• Realistic Validation: Tests real-world interaction between components.

Disadvantages

• Complexity and Cost: Setting up integration tests can require significant effort to configure dependencies and environments, and issues uncovered at this stage can be costlier to address than those identified in unit tests.
• Performance: Slower than unit tests.

Example

Testing the interaction between a payment gateway API and an internal order management system.

Challenges and Mitigation

Setting up integration tests can be complex. Use continuous integration (CI) pipelines to automate the setup and teardown of environments.

Building and Maintaining a Regression Pack

Integration tests play a critical role in a regression test pack, which ensures that new code changes do not break existing functionality. Best practices include:

• Automating the regression pack as part of the CI/CD pipeline for continuous feedback.
• Periodically reviewing and updating the regression tests to keep them relevant.
• Prioritising critical workflows in the regression suite to minimise maintenance costs while maximising impact.

E2E Testing: The Ultimate User Experience Validator

What is E2E Testing?

End-to-end testing replicates real user scenarios to validate the entire application workflow from start to finish. It ensures that the system behaves as expected from the perspective of the end user.

Tools and Frameworks

Popular tools for E2E testing include:

• Selenium: A widely used framework for automating web browsers.
• Cypress: Known for its developer-friendly features and debugging capabilities.
• Playwright: A modern E2E testing tool supporting cross-browser testing.

Use Cases

E2E testing shines in scenarios like:

• Verifying user authentication flows, such as login and registration.
• Ensuring seamless transaction workflows, such as shopping cart and checkout processes.
• Testing visual interfaces (UI/UX) to validate overall application design and responsiveness.

Advantages

• Comprehensive Coverage: Validates the system’s functionality and reliability from the user’s perspective.
• High Confidence: Provides assurance that the application behaves as intended in real-world scenarios.

Disadvantages

• Slow Execution and High Cost: Takes longer to execute compared to other testing methods and is the most expensive phase due to the need for realistic setups and broad scope.
• Maintenance Overhead: Challenging to maintain for applications undergoing frequent updates.

Example

Simulating the complete booking process for a flight reservation system, including search, seat selection, and payment.

Challenges and Mitigation

E2E tests can become brittle with frequent UI changes. To reduce maintenance overhead:

• Modularise tests and reuse components.
• Regularly review E2E tests for relevance and adapt them to evolving workflows.

Striking the Right Balance in Your Testing Strategy

Layering Your Tests

The pyramid visual highlights the importance of balancing test types:

• Unit Tests (70%): The foundation, covering the majority of the codebase.
• Integration Tests (20%): Validating interactions between components.
• E2E Tests (10%): Ensuring critical workflows function as expected.

Decision Criteria

• Unit Tests: Ideal for projects with rapid iterations or a focus on algorithmic correctness.
• Integration Tests: Crucial for API-heavy architectures or systems with multiple dependencies.
• E2E Tests: Essential for applications where UI/UX is a critical success factor.

Automation Strategies

Automate testing layers wherever possible. Use CI/CD pipelines to integrate testing into the development workflow for continuous feedback.

Best Practices for Testing

To create an efficient and maintainable testing strategy, follow these best practices:

• Write testable code with clear boundaries and separation of concerns.
• Keep tests independent and reproducible.
• Use descriptive names for tests to improve readability.
• Regularly review and refactor tests to ensure relevance as the application evolves.

Conclusion

Each type of testing — unit, integration, and E2E — serves a unique purpose in ensuring software quality. Unit testing provides quick feedback and focuses on individual components, while integration testing ensures that interconnected parts of the system work together. Finally, E2E testing replicates user workflows to validate the entire application’s functionality.

By leveraging these testing methods in a layered approach, you can achieve a balance between speed, coverage, confidence and cost. Whether you are a developer, QA engineer, or software architect, understanding and applying these testing practices will help you build robust, reliable, and user-friendly applications.

A Comprehensive Guide to Software Testing: Unit, Integration, and E2E Testing Explained was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Defensive programming is a coding method where engineers anticipate potential issues, errors or misuse of their code and write software to handle these situations proactively.

It aims to improve the overall quality, stability and security of code by anticipating and handling potential errors or unexpected scenarios or failures, even if the input or environment is not ideal.

Let’s use an example to explain further. I will be using JavaScript as my example language, but the concepts apply to any software language.

Example 1: Simple calculation

Let’s take an example of a simple number calculation.

function calc(a, b) {
  return a+b;
}

calc(1,1) // returns 2

Tested, no issues here so it’s ready for production right? Wrong!

What happens when the input is not an integer, but a text or any other data type? Will it return the expected value?

calc(a,1) // Uncaught ReferenceError: a is not defined

This piece of code is not robust at all. We need to ensure the values are numbers and not any other datatype.

function calc(a, b) {
  // Check if the input values are numbers
  if (isNaN(a) || isNaN(b)) {
    return "You need to enter a number";
  } 
  
  return a+b;
}

This simple condition to check if the input values are numbers will now at least ensure the code is considering non-numerical input values and returning an appropriate message for that condition.

Example 2: Selecting an element

Let’s say you wish to get the value of an element on the page with the “foo” identifier.

const elFoo = document.querySelector("[data-js=foo]");

elFoo.innerText;

If the page has the element with that identifier with a value, it will work with no issues, however, what happens when the page running this code does not have that element?

Uncaught TypeError: Cannot read properties of null (reading 'value')

It will return the above error and the remainder of the JavaScript code will not execute.

The issue here is that the code was written assuming that element will always exist. A defensive programming approach will ensure the code is written such that even if the variable (in our case the element) does not exist, it will not break the execution of the code.

Simply adding a condition to check that the element exists before continuing is defensive programming.

if (elFoo) {
  return elFoo.innerText || “Placeholder Text”;
}

Key aspects of defensive programming

1. Input validation: Thoroughly validating, verifying and sanitising all input data to prevent errors or security vulnerabilities.

2. Error handling: Properly managing errors using try/catch blocks, exception handling or other mechanisms to ensure that the code can gracefully recover from or handle errors.

3. Avoid assumptions: Not assuming that functions, inputs, or other parts of the system will always behave as expected. This includes being prepared for edge cases or incorrect usage and catching logical errors early in the development process.

4. Secure coding practices: Implementing security measures to protect against common vulnerabilities like buffer overflows, SQL injection or cross-site scripting.

5. Logging and monitoring: Incorporating comprehensive logging to aid in debugging and identifying issues in production.

6. Fail-safe defaults: Designing systems to fail safely if something goes wrong, ensuring that critical systems remain secure or operational in a limited capacity.

7. Code clarity: Writing clear, maintainable code that is easy to understand and modify. This reduces the risk of future bugs when the code is updated or extended.

8. Defensive data structures: Using immutable objects or implementing checks to maintain data integrity.

9. Immutability and side effects: Limiting the use of global variables or mutable states, which can cause unexpected behaviour if not properly managed.

10. Thorough testing: Employing unit tests, integration tests, and edge case scenarios to verify code behaviour.

By following defensive programming practices, developers can build systems that are more resilient and less prone to failure in unexpected circumstances. It makes the software safe, secure and robust.

Importance of Defensive Programming was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In today’s news world, timely and accurate information is very critical and breaking news and frequent updates are vital for media companies to maintain credibility and readership.

Thus, measuring the news publishing velocity, which is how quickly the publish-ready content is made available to our readers, and content freshness, which means that the most up-to-date content is always displayed, have become increasingly critical in today’s fast-paced world.

Why Do Content Freshness and Publication Speed Matter?

By tracking how quickly stories transition from publication-ready to accessible and ensuring the latest content is delivered, we enhance engagement, identify opportunities to improve operational efficiency and remain competitive. These advantages can be summarized as follows:

• Boosting Visibility in the Media Landscape: Being the first to report on significant events enhances The Telegraph’s position as a leading news source.
• Building Reader Trust: Timely and transparent updates on new facts or corrections demonstrate journalistic integrity, strengthening trust and audience loyalty.
• Enhanced SEO rankings and search engine trust: Consistent updates signal to search engines that the site is active and authoritative, outperforming stagnant content and drawing in more readers.
• Dynamic site experience and Increased engagement: Frequent updates encourage readers to return, and create a vibrant, immediate feel, keeping the site lively and engaging.

How Does Content Publication Work?

To understand The Telegraph’s approach to addressing this challenge, it’s important first to provide some context on the underlying technology. Since Telegraph’s content is managed via Adobe’s CMS: Adobe Experience Manager as a Cloud Service (AEMaaCS), we need to explore how content propagates through the various layers of the system to ensure seamless publication and distribution.

The Lifecycle of Article Publication

1. Telegraph Authoring
   The content creation process begins with our custom Telegraph Authoring tool. This provides an easy-to-use interface for editors to draft, revise, and finalise the article. The content itself is persisted in the Author layer of Adobe Experience Manager (AEM). After the content is crafted, it undergoes editorial workflows, including approvals and revisions, ensuring the quality and accuracy of the article. Once approved, it’s ready for publication.
2. AEM Author:
   This is where the article content and associated assets are stored. In Adobe Experience Manager, the Author instance hosts the content repository. In our case, it also acts as the intermediary that receives publish requests from the Telegraph Authoring and initiates a replication request to publish the content to the Publisher.
3. Publisher:
   After the author finalises the content and hits the “Publish” button, the published article gets replicated from the authoring environment to the Publisher instance. In other words, the Publisher instance in AEM handles the initial steps of making the content live and ensuring it becomes accessible to further layers.
4. Dispatcher:
   The Dispatcher caches content to reduce server load and speed up delivery to our readers. It ensures that frequently accessed articles are quickly served, reducing the load on the backend and improving readers' response times. In addition, this is where the resolution of Server-Side Includes (SSI) takes place, which we will mention later.
5. Fastly CDN:
   It acts as a layer that further optimises the speed and scalability of content delivery by caching it at edge servers close to our readers. This reduces latency and ensures that readers, regardless of their location, experience fast load times for the latest articles.
6. External CDN:
   To enhance global reach and reliability, the content is also distributed through another external CDN. This provides an additional layer of global distribution, ensuring redundancy, fast loading speeds, and protection from traffic spikes. Together with Fastly, it ensures the smooth, fast, and secure delivery of content to readers worldwide.

In summary, when an article has passed all editorial workflows, including revisions and approvals, and the ‘Publish’ button is clicked in our content authoring platform, the content gets distributed across various layers of the system. Each of these layers works in harmony to ensure that articles are published efficiently, cached appropriately, and delivered quickly to a global audience, maintaining both performance and reliability.

Measuring Publishing Velocity

We follow a multifaceted approach that starts with the implementation of timestamp-based versioning to ensure each change is accurately recorded and tracked over time. This enables us to maintain a clear history of updates. Once a page is published, we utilise an event-driven approach that checks the live version of the page to confirm it reflects the most recent changes. The sections below provide more information along with the key challenges of this approach.

Timestamp-based Versioning

At a high level, to distinguish the latest content from previous versions on the client side, we mark each update with a timestamp in the article’s HTML using a predefined format. For example, if we publish an article about a breaking political event on November 1, 2024, at 3:00 PM, and later update it at 3:15 PM, our system ensures that the updated timestamp is reflected in the article’s markup, signalling the change.

Event-driven Approach

We leverage Google Cloud Platform (GCP) offerings, utilising a Pub/Sub topic to hold the publish event messages alongside a serverless Cloud Function that acts as our monitoring bot. This process works as follows:

1. Listening to Publish Events: The Cloud Function subscribes to the Pub/Sub topic to receive notifications whenever new articles are published.
2. Making Requests: Upon receiving a publish event, the Cloud Function sends HTTP requests to the published article page, simulating how a real reader would access it.
3. Checking Versions: The function verifies whether the article has been updated by checking the latest timestamp version and matching it with the expected one received in the publish event.
4. Monitoring: The function logs metrics such as publish times, errors, and any discrepancies. These are then reported to our Datadog dashboard for real-time monitoring and analysis.

Key Challenges

While this approach may seem straightforward, it is not without its challenges. The main ones are:

1. Impact on SEO: Improperly managed timestamps in the article’s HTML, which may already contain other metadata such as the latest modification date and structure data, could lead to confusion about the content’s recency and negatively affect search engine optimisation (SEO).
2. Outdated Cache: As the CDNs may cache the article page, this can result in our monitoring bot seeing outdated content instead of fresher content at the previous layers.
3. Content Propagation Bottlenecks: In our multi-layered system, content propagation can encounter bottlenecks at various stages. As a result, the absence of the latest updates at the outer layer provides limited insight into where the content update may have stalled along its journey to the final destination.
4. Independently Updatable SSI Content: Some pages exist within a parent-child structure, where the publicly available parent page includes references to one or more child pages via server-side inclusion (SSI). For example, a Live Blog may have references to Posts, though Posts are not independently accessible. These posts can be updated and republished separately from the parent. However, we want to ensure that the publicly visible parent page reflects the latest content from its child pages as well as measures how quickly it integrates these updates.

Addressing the challenges

To tackle these challenges, we must carefully evaluate both our use of timestamp-based versioning and the effectiveness of our event-driven approach.

1. SEO Protection: Embed Timestamps in Comments
To mitigate any potential issues when adding metadata related to the page’s recency, we insert the timestamp information within an HTML comment. Comments are ignored by browsers and generally disregarded by most web crawlers. This ensures that they do not interfere with page rendering or impact search engine optimisation.

<!DOCTYPE html>
<html lang="en">
   <!-- <timestamp-key>: <timestamp-value> -->
</html>

2. Outdated Cache: Cache Bypass with Allowlists
Bypassing cache via allowlists is a strategy used in Content Delivery Networks to allow specific requests or clients to bypass the cache entirely based on defined criteria. This is typically done by creating an “allowlist” of conditions, such as IP addresses, user agents, or request headers, that should always fetch fresh content directly from the origin server instead of the cached version. This approach ensures our Cloud Function can make requests while ensuring the latest version of the content is fetched, bypassing the cache.

3. Content Propagation Bottleneck Detection: Granular Hop Times
By configuring the requests in our Cloud Function to bypass CDN caches and target deeper layers directly, we can make two types of requests: one that bypasses the cache and one that doesn’t. This allows us to measure content propagation times both before and after the cache, providing insights into latency at each stage. We’ve identified several key timing points for tracking. These include:

a. time-at-author: the timestamp when the authoring instance initiates a publish event.

b. time-at-pubsub: the timestamp when the Cloud Function is triggered upon the publish event’s reception by Pub/Sub.

c. time-to-origin: the time it took for the content to be available at Fastly.

d. time-to-edge: the time it takes for content to reach the edge CDN.

By adding these time markers in addition to the total publish time, we can gain valuable insights into the latency at each stage of the delivery pipeline if we face any issues during our monitoring of the publish velocity.

4. Independently Updatable SSI Content: Retrieve Parent Page When Child Pages are Published

When a Post page (a child page) is published, we retrieve the URL of its parent Blog page and create a request for that URL. Then, we ensure the parent page displays the latest updates from its associated child pages by looking at the timestamp marker of the latest updated child page within the parent page. Eventually, we compute the speed of child page updates by measuring the time elapsed between the publish event (see Figure 3) and when the updates are reflected on the parent page.

Leveraging Real-Time Data in Datadog

All metrics gathered by the Cloud Function are visualized in real time on our Datadog dashboards. We leverage Datadog to define and monitor SLIs (Service Level Indicators) based on the publish time calculations.

Publish time thresholds differ by content type. For example, for live articles (such as reporting football matches, elections, and breaking news), the publish time must be almost instant to ensure updates are available to readers promptly. On the other hand, for regular articles, slightly more relaxed thresholds are acceptable as the content doesn’t require frequent updates.

Alerts are triggered if any SLI falls below the defined thresholds, and our on-call is notified immediately. This ensures rapid identification and resolution of issues, maintaining the performance standards of our publishing workflow.

Summary

The publication process leverages Adobe Experience Manager for content creation and distribution, ensuring efficient and timely delivery to readers through caching and content delivery networks.

In this multi-layered system, we employ timestamp-based versioning and an event-driven approach to track updates across each layer, identifying key measurement points to detect potential bottlenecks. This ensures our metrics accurately reflect publish velocity and workflow efficiency, enabling us to proactively address any potential issues.

News in Real Time: The Telegraph’s Approach to Publishing Velocity and Content Freshness was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Just a simple service

When object cache is mentioned, engineers tend to think of static assets. Images, videos, text. All of which can be combined to make reports, stories, opinion pieces and much more. Put that all together and it is easy to see an online newspaper on the other end.

As a news organisation, the Telegraph relies on layers upon layers of caches to serve content all around the world.

As several people have observed (including Martin Fowler), cache invalidation is a real challenge. In the Telegraph’s context, certain actions require the cache to be invalidated on demand. For that purpose, the Platforms team constructed a cache flush service to clear the cache of several layers on demand.

Seems simple enough; a service that clears caches on request. Yet the years have not been kind to it. Feature creep, dead code and quick patches took their toll and maintaining the service was becoming a real chore.

Tech debt is never fun

A rolling stone gathers no moss, goes the old proverb, and there is no better saying to express the need for an organisation to be fast-moving, lest it be left behind with no hope of catching up to competitors and, more importantly, customers. The Telegraph is no different in that regard. Just as the torrent of news never stops, the organisation needs to be on the cutting edge of technology to secure and improve upon the competitive advantage that makes this news organisation unique and successful.

Striving to be ahead of the current has great benefits, but there are a lot of challenges as well, readily apparent for anyone who has worked in technology for a while. One of the most glaring ones is the accumulation of technical debt.

Every engineer who has been around the block knows the feeling. The next task arrives and there is a need to improve upon a project with no commit in years. Documentation is non existent. A defeated sigh, a couple hours (or days) of struggling and the new feature is there. The feature is shipped, and the project is forgotten for a while. A new feature request comes. Now, the changes take a bit longer. A patch here and there, no reason for a full solution now. Then, a new feature request.

A couple of feature requests down the line, patch upon quick hack, it becomes literally impossible to modify the functionality without re-writing the whole thing. And new feature requests keep coming.

In hindsight, we should …

One of the benefits of working at the Telegraph is that when such a situation is detected, it is dealt with as a serious concern. As such, engineers are encouraged to re-evaluate architectural decisions of the past and, more importantly, request resources to deal with those issues.

Of course, no organisation wants to be bogged down by the constant modernisation of the technical stack, which is truly an exercise in futility; technical debt is part of the reality of software development. However, great organisations take into account concerns of such nature and, when there is an opportunity, give engineers the go-ahead to fix those issues and bring the technical stack up to speed.

A fresh technical solution, time-boxed, with undeniable benefits, is hard to turn down.

Examining the old

Re-architecture starts, as one would expect, by examining the service in question.

One must approach the old service with respect, trying to understand what made that service successful in the first place. It is important to keep the good parts and avoid the not-so-good ones.

A RESTful Springboot microservice with a MySQL database is as standard a microservice as you can get. But even that basic design needs to be re-examined.

As clients communicate with the service using REST, that should be expected to remain the same for the new design as well. After all, communication contracts should only change if there is an absolute need for it.

Regarding the MySQL database, examining the service code reveals that the need for it is the communication with 3rd-party cache providers. Rate limits are always a concern; hence, there is a need to keep a steady rate of requests. To achieve that, the database works as a queue, with the service polling the database for stored jobs. From this, we can infer two points. One, rate limits are a concern and two, some clients do not mind asynchronous communication.

Other issues become apparent if one is inclined to observe the operation of the service. The daily requests are not a lot, 30 to 200; however, the service operates 24/7, consuming resources and accumulating costs. As such, the MySQL database queue remains empty for most of the time, with the maximum amount of queue size a measly 2 items. Maintaining a full MySQL database for this load seems like a waste of resources.

Infrastructure-wise, the service itself is deployed on a Kubernetes cluster, with the database deployed using the CloudSQL product. The Platforms team is doing an excellent job in maintaining an operational, up-to-date cluster, but as any DevOps team would attest, the inherent complexity of maintaining a Kubernetes cluster can be frustrating at times. Avoiding that complexity would be ideal.

To recap:
+ RESTFull
+ Rate Limits
+ Async
- 24/7 Operation
- Database Queue
- Infrastructure complexity

Who needs servers?

An important part of the day-to-day operations at the Telegraph is the Google Cloud Platform. As the cloud provider of preference, many of our services and operations rely on Google's building blocks to streamline development and operations.

Evaluating the offerings available, we realised we could leverage the serverless offerings to turn our requirements into a modern, low-cost architecture.

A new perspective

The low rate of daily requests meant that the service did not need to run 24/7. For that purpose, the team decided to turn the deployment into a Cloud Run service that scales down to zero. In that way, compute resources would only be used when needed. As an added benefit, it meant no more struggling with Kubernetes major version upgrades.

However, going that route meant that the Springboot application needed reconsidering. Springboot has a notoriously slow startup time and the JVM is not well suited to the strict cold-start requirements that the scaling down to zero paradigm requires. The decision was made to re-write the service in Python using Fast API since the team is experienced in Python development and the cold-start and response times are well within the acceptable limits. Also, re-writing the service would give the team the opportunity to simplify the software design and do away with any functionality that had outlived its use.

The next challenge to tackle was the rate limits of the 3rd-party APIs. As the investigation of the original service showed, the daily requests are not nearly close to reaching the rate limits, but the design needs to be secure against a burst of requests coming in. Such a scenario can easily happen on a breaking news day.

Given that clients are fine with asynchronous requests, meaning they can request a clear cache and not wait for the result, an asynchronous part can be added to the design that takes care of the rate-limiting issue. To avoid using a database as the old architecture did, an offering called Cloud Tasks was used to hold a processing queue. Cloud Tasks acts as a buffer queue, where the rate of delivery can be defined, along with a default behaviour to slow down requests when a 429 Too many requests error surfaces. All that remained was a small additional service that took care of the scheduling in the queue, which would also be implemented in Cloud Run using Fast API.

Finally, clients should have the option of calling the cache clear service directly or going through the asynchronous path. For that purpose, both services would be exposed using Load Balancers configured to handle TLS and drop unauthenticated traffic.

After careful consideration of the points above, here is the resulting design.

Out with the old one, in with the new

After the new design was reviewed and implemented, the implementation was moved to production for a dry run. When that was verified, the proxy was pointed to the new services, diverting the traffic to the new infrastructure. Monitoring using Cloud Logging, we verified that everything was operating as expected.

As expected, the requests are scarce, proving that the scale to zero Cloud Run design was an fitting choice.

The Cloud Tasks queue demonstrates that our assumption of 1–2 concurrent tasks was accurate and easily manageable.

With the new design, we were able to:

• Reduce the cost of operation for the service by scaling compute to zero
• Reduce the maintenance overhead of the service by leveraging Google-managed infrastructure
• Shed most of the technical debt of the old service

Moving forward, we will consider all opportunities that the serverless paradigm offers to design architectures that are more reliable, modern and cost-effective, all the while keeping an eye out for old services in need of a new coat of paint.

Georgios Makkas is a Senior Platform Engineer at The Telegraph Media Group

Leveraging serverless Google Cloud Platform features for fun and profit was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to set up a dead-letter for an Eventarc trigger using gcloud CLI and track your serverless exceptions

Welcome to the first of a 2-part series on how you can set up dead-letter policies for Eventarc triggers on serverless applications. This time we look at creating one using gcloud commands. Next time, we will guide you on doing the same using Terraform.

Serverless has presented a big opportunity in The Telegraph’s quest for more efficient and environmentally-friendly services. Instead of a server or Kubernetes-based microservice running 24 hours a day, we can leverage GCP (Google Cloud Platform) Cloud Functions that only execute code for a small fraction of that time. One of the crucial pieces of building out a reliable serverless-based service is how and when you pull the trigger on the code executions and what happens when the code throws an exception.

Enter Eventarc triggers, which offer breadth and depth. Breadth in what can act as a trigger. Do you need to run code based on logging events, a pubsub topic, write events on a bucket, or changes to specific file or filetypes? They can all be triggers. And depth in what they can execute. Eventarc triggers can fire up Cloud Run containers or Cloud Functions, giving you the choice of container-based or serverless code executions that are suitable for a great deal of tasks.

But like many out-of-the-box solutions, Eventarc triggers are also a closed box with little control over them. In this blog, we’ll explore how to configure the Eventarc trigger’s behaviour in a way that gives us more control.

At its core, Eventarc events consist of a simple pubsub architecture. If a trigger is set to fire on a bucket object write event, it would essentially send a message to its pubsub topic, and a subscriber would consume the message and act on it.

Every pubsub subscription can be configured to have a dead-letter policy. Messages are sent to a dead-letter topic if a number of delivery attempts are exhausted with no acknowledgement from the subscription. This provides us with a way of exposing and managing failures of the code to complete.

Below, we will show you how to create your dead-letter policy using gcloud CLI.

The initial setup

Prerequisites

To perform the following steps, you will need to ensure the following:

• You have the gcloud CLI installed.
• You have a GCP account and have configured its access to a GCP project with gcloud CLI.
• Your account must have the requisite permissions to create the resources detailed below in your project.

Getting started

In thefunction-deploy branch of this repository, you will find instructions on deploying a simple cloud function. The cloud function is set to trigger a pubsub message to a topic and will print a greeting message containing the message.

If the message="fail", the code is set to raise an exception, which will cause the function to exit ungracefully.

Create the pubsub topic and function

## create pubsub topic
gcloud pubsub topics create demo-pubsub-topic

## create cloud function
gcloud functions deploy demo-pubsub-function \
    --gen2 \
    --runtime=python311 \
    --region=europe-west2 \
    --source=./src \
    --entry-point=subscribe \
    --trigger-topic=demo-pubsub-topic

Trigger function (print greeting)

gcloud pubsub topics publish demo-pubsub-topic --message="World"

This will cause the function to execute and print the greeting Hello World, and we can verify this by looking at function logs and running the following command.

gcloud beta functions logs read demo-pubsub-function — region=europe-west2 — gen2

Trigger function (fail)

gcloud pubsub topics publish demo-pubsub-topic --message="fail"

This message will cause the function code to throw an exception, which will cause the function to exit ungracefully. As before, we can verify this by checking the function logs.

Enable retry

At this point, the function is set to not retry upon failure. We can change that by adding a — retry flag to our function deploy command:

gcloud functions deploy demo-pubsub-function \
    --gen2 \
    --runtime=python311 \
    --region=europe-west2 \
    --source=./src \
    --entry-point=subscribe \
    --trigger-topic=demo-pubsub-topic \
    --retry

If you now open your GCP console and check the function information, and go to the trigger tab, you’d see that the retry-on-failure is enabled.

Also, if you head over to the Pubsub page and click on the topic we created, you will see that it has a phantom pubsub subscription attached to it. We never created that subscription; instead, the gcloud functions deploy command took the liberty of creating and attaching that subscription to the topic.

By clicking on the subscription and then the Details tab, you’d see that the dead-lettering is disabled on that, but back-off and retry are enabled because we added the--retry flag.

Dead-letter Setup

First thing we’d need is a pair of pubsub topic and subscription to use as dead-letter. A dead-letter topic is just another pubsub topic.

Create a dead-letter topic/subscription

The following commands will get us the pubsub topic and subscription.

gcloud pubsub topics create demo-deadletter-topic
gcloud pubsub subscriptions create demo-deadletter-subscription --topic=demo-deadletter-topic

Attaching the dead-letter topic

From the UI, it’s easy for us to get the Eventarc subscription name, but where’s the fun in that? We will instead use the gcloud command to get the subscription id and then use that. Since it’s not straightforward, we’d have to fetch this information in two different steps:

• Get the function info and grab the trigger name from that
• Using the trigger name, get trigger info which contains the subscription name

The following commands will help us achieve that:

trigger_name=$(gcloud functions describe demo-pubsub-function --region=europe-west2 --format='value(eventTrigger.trigger)')
trigger_subscription=$(gcloud eventarc triggers describe $trigger_name --location europe-west2 --format='value(transport.pubsub.subscription)')

We can run echo $trigger_subscription to confirm whether we have the correct subscription id.

Update subscription to attach the dead-letter

Let's update the Eventarc subscription and attach the dead-letter topic to it.

gcloud pubsub subscriptions update $trigger_subscription \
    --dead-letter-topic=demo-deadletter-topic

Grant required permissions to the project service account

One last and necessary step is to give the GCP project’s pubsub service account the roles/pubsub.subscribe permission on the Eventarc subscription and roles/pubsub.publisher to the dead-letter publish. This is very important and it will ensure the project service account has all the permissions required to be able to read the failed messages and forward them to the dead-letter topic.

The project’s pubsub service account email format is as below:
service-<GCP_PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com

For example, if your project number is 1234567890, the service account would be service-1234567890@gcp-sa-pubsub.iam.gserviceaccount.com

We can give the service account the required permissions by running:

## assign pubsub subscriber role to read messages from trigger subscription
gcloud pubsub subscriptions add-iam-policy-binding $trigger_subscription \
    --member="serviceAccount:service-<GCP_PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com" \
    --role="roles/pubsub.subscriber"

## assign pubsub publisher role to allow publishing the dead letter topic
gcloud pubsub topics add-iam-policy-binding demo-deadletter-topic \
    --member="serviceAccount:service-<GCP_PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com" \
    --role="roles/pubsub.publisher"

That’s it, we’re done! We can confirm that by checking our Eventarc subscription.

We can also confirm that the entire setup is configured correctly by checking the DEAD LETTERING tab.

Test the dead-letter functionality

Send a message to the pubsub topic that will make the function code raise an exception.

gcloud pubsub topics publish demo-pubsub-topic --message="fail"

Check pubsub dead-letter for failed messages

Due to the retry and backoff mechanism, this might take a couple of minutes to come through.

gcloud pubsub subscriptions pull demo-deadletter-subscription

The final code can be found in the main branch of this repo.

Stay tuned for part 2 of our guide, where we will walk you through how to configure a dead-letter to Eventarc using Terraform

Irfan Durrani is a lead platform engineer at The Telegraph. You can follow him on LinkedIn. You can also follow The Telegraph Engineering team on LinkedIn.

How to set up a dead-letter for an Eventarc trigger using gcloud CLI and track your serverless… was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Kubernetes puts the mantra of “fail fast, recover faster” into practice: one thing fails, and another seamlessly pops up to replace it. But what happens when that guiding principle works against it? We’d like to share one such scenario we encountered in GKE, our findings and how we overcame it.

While we first encountered this issue some 2 years ago, I have seen it crop up on StackOverflow, so it’s not a rare occurrence.

I raised it in the Kubernetes project, and have also seen it referenced in other issues, too.

Symptoms

One day we noticed that Jobs and CronJobs in our cluster were acting strangely; multiple Jobs had spawned and sat there for more than an hour without creating any pods.

Digging into the Problem

Investigating other jobs, we found that a single CronJob in one of our namespaces was responsible for creating more than 900 pods. These pods had completed but had not been removed.

The CronJob was scheduled to run every minute, and the definition of the CronJob had valid values for the history — sensible values for .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit were set. Even if they weren’t, the defaults would (or should) be used.

Why did we have over 900 cron pods, and why weren’t they being cleaned up upon completion? Just in case the number of pods were causing problems, we cleared out the completed pods:

kubectl delete pods -n {namespace} $(kubectl get pods -n {namespace} | grep Completed | awk '{print $1}' | xargs)

Even after that, new jobs weren’t spawning pods. In fact, more CronJob pods were appearing in this namespace. So we disabled (suspended) theCronJob

kubectl patch cronjobs -n {namespace} {cronjob-name} -p '{"spec" : {"suspend" : true }}'

That also didn’t help; pods were still being generated. Which is weird — why is a CronJob still spawning pods, even when it’s suspended?

The Lightbulb Moment

CronJobs generate Job objects when their schedules trigger. On checking the Job objects, we found more than 3,000 Job objects; a lot more than there should for something that runs once a minute. So we deleted all the CronJob related Job objects

kubectl delete job -n {namespace} $(kubectl get jobs -n {namespace} | grep {cronjob-name} | awk '{print $1}' | xargs)

This reduced the pod count, but did not help us determine why the other Job objects were not spawning pods.

Bringing In Reinforcements

We raised a support ticket with Google, who initially sent us this logs snippet (redacted):

2020-08-05 10:05:06.555 CEST - Job is created

2020-08-05 11:21:16.546 CEST - Pod is created

2020-08-05 11:21:16.569 CEST - Pod (XXXXXXX) is bound to node

2020-08-05 11:24:11.069 CEST - Pod is deleted

2020-08-05 12:45:47.940 CEST - Job is created

2020-08-05 12:57:22.386 CEST - Pod is created

2020-08-05 12:57:22.401 CEST - Pod (XXXXXXX) is bound to node

Spot the problem? The time between “Job is created” and “Pod is created” is about 80 minutes in the first case, and 12 minutes in the second one. It took 80 minutes for the Pod to be spawned.

It All Becomes So Clear…

This is where it dawned on me about what was possibly going on.

• The CronJob spawned a Job object. It tried to spawn a pod, and that took a significant amount of time, far more than the 1 minute between runs.
• The next cycle, the CronJob looks to see if it has a running pod due to the .spec.concurrencyPolicy value.
• The CronJob does not find a running pod so generates another Job object, which also gets stuck waiting for pod generation.
• And ad infinitum.

Each time, a new Job gets added, gets stuck waiting for pod generation for an abnormally long time, which causes another Job to be added to the namespace which also gets stuck…

Eventually, the pod will generate but by then there’s a backlog of Jobs, meaning that even if I suspended the CronJob, it won’t have any effect until the Jobs in the backlog are cleared or deleted.

The Cause

Google investigated further, and found the culprit:

Failed calling webhook, failing open www.up9.com: failed calling webhook "www.up9.com": Post https://up9-sidecar-injector-prod.up9.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

We were testing up9 and this configuration was using a webhook, so it looked like a misbehaving webhook was causing this problem. We removed the webhook and the webhookconfigs, and everything started working again.

Lessons Learnt

So where does this leave us? Well, a few thoughts:

• A misbehaving/misconfigured webhook can cause a snowball effect in the cluster, causing multiple runs of a single CronJob without cleanup — successfulJobsHistoryLimit and failedJobsHistoryLimit values are seemingly ignored.
• This could break systems where the CronJob is supposed to be run mutually exclusively, since the delay in pod generation could allow two cron pods to spawn together, even though the CronJob has a concurrencyPolicy set as Forbid.
• If someone managed (whether intentionally or maliciously) to install a webhook that causes this pod spawning delay, and then adds a CronJob that runs once a minute — and then maliciously crafts the job to never finish, this snowball effect will cause the cluster to run out of resources and/or scale up nodes forever or until it hits the max allowed by your configuration. Fortunately, this is now known by the Kubernetes team.

Johnny Ooi is a Senior Site Reliability Engineer at The Telegraph. You can follow him @Blenderfox.

The “Snowball” Effect in Kubernetes was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

When old news is bad news

A major, multi-week event was underway when a call came in from Editorial: it looked like an old version of a story was appearing on a syndication partner’s feed, but which version?

A bit of context. At The Telegraph, as well as publishing news across in-house print and digital platforms, syndication means that our stories appear in publications around the world. A vital job of the Platforms team is to make sure that our articles reach our syndication partners safely. With stories evolving on a 24-hour basis, it’s also imperative that the content we are sending out keeps pace with our own reporting.

On receiving the call from Editorial, we found 2 things; firstly, the article appeared to be out of sync, and secondly, getting to the error’s root cause lay behind a myriad of logs and we could not provide immediate answers. At the same time, this scenario highlighted the need for an altogether better solution. It marked the birth of the Content Status project.

A simple event-driven solution

Keen to move on from a web of log entries, we decided to keep the design simple. Events would be emitted from each microservice with metadata information about the articles being published to our partners. Those events would be captured by a service that would store them in the proper format in a storage space. The same service would provide the relevant endpoints to expose the information.

Having worked with Google Cloud’s Pub/Sub before, we felt confident that it would serve as an event queue that would allow us to decouple the event processing from the article publishing. Such a decoupling was crucial; we did not want to impede the high performance of the microservices serving our syndication partners.

For storage, we opted for an Elasticsearch cluster. With the amount of events expected, Elasticsearch could scale without issue, handling this considerable amount of documents and at the same time providing the search capabilities that Elasticsearch is famous for.

As for the event processing service, we went with the familiar Springboot project, deployed in a Kubernetes cluster. The horizontal scaling capabilities native in Kubernetes would allow us to handle the message load (or so we thought at that moment. More on that later…).

To expose the information, we decided to use an internal tool called the Toolbox. It is an in-house solution used to display all kinds of information regarding our articles, as well as provide several functionalities for different departments.

The following is the finalised system design.

To accommodate the publishing of events, we created a simple library and put it to use in our microservices. With a simple interface, information about article processing could be sent along with any useful metadata (success/failure, error message, etc).

All of the events were published in one particular topic, to which the event processing microservice was subscribed. Google Cloud Console provides excellent visibility when it comes to the amount of messages published, acknowledged, failed and others.

Too much information? Multithreading to the rescue

All seemed ready, so we flipped the switch.

At that moment, a major issue became apparent.

The load on the event processing microservice was huge. Even multiple replicas of the service were not able to handle the amount of events being published. Production conditions can be a very unpleasant surprise.

We decided to increase the number of processing threads. For those of you familiar with multithreading, you now realise that the number of threads is not as apparent. To play it safe, we simulated an approximation of the traffic on our staging environment and were able to find the golden ratio.

After the change, the number of unacknowledged events started to drop sharply. Soon, the service was operating without issues.

The answers we need in one place

With the service and UI implemented, tracking an article’s syndication history became a breeze. Using an article Id, under the History tab, users can easily track what has happened.

Using the tool, we are able to identify:

• If the article was published to one of our Syndication partners
• The latest version of the article that was published
• An initial indication of any failures
• Immediate feedback on the state of an article regarding our Syndication partners

Now, if Editorial or a Syndication partner needs to know what happened to an article, we can provide a much quicker and more precise answer. On a happy side-note, we are much more able to verify that a release is not affecting article processing.

So from the pain of trawling through logs, we have managed to implement a system that now provides at-a-glance clarity and reassurance for The Telegraph and its partners, while giving any insights for any potential errors down the line.

Georgios Makkas is a Platform Engineer at The Telegraph

Taking the grind out of article history investigation through Pub/Sub centered architecture was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

How The Telegraph built a smoother release management solution that keeps pace with a busier schedule of changes, while ensuring accountability every step of the way

Overview
Introduction
How it works
The user journey in Slack
Automated record-keeping in JIRA
Adding Github Action functionality
The results
Useful links

Microservices sit at the heart of what and how we build software at The Telegraph. This has made for improvements to performance, resilience and the maintenance of our applications. But as the volume of services swells over time, so too does the number of releases we need to make. Every month we perform dozens of releases to our systems. Our dilemma? Ensuring that these changes are properly reviewed, approved and documented, whilst removing any unnecessary friction.

The right solution for us laid in tools that were already at our disposal. The Telegraph has been using Slack for several years and its Bolt SDKs make it relatively easy to write custom applications. Likewise, JIRA has long been our primary means of accounting for work, including change requests.

We found that, by building an API to handle the flow of information and requests between users, Slack and JIRA, we could seek and co-ordinate releases in a more convenient and more visible manner. In addition, we were able to take advantage of Github Actions to add further improvements to the process when making changes to project repositories.

What follows is truly a joint effort; this project’s creation and continual improvement has been the work of the whole team coming together and contributing ideas, to forge a system that meets our needs brilliantly.

How it works

This diagram tries to convey the lifecycle of a change request using our system. I have numbered the steps to help lay out the basic chronology.

The user journey in Slack

One of the key functions of our API is to return json strings to Slack that it uses to populate the modal screens for our request process, as well as parse the data being sent over Slack and use it to create and update tickets in Jira. This being the case, I will illustrate just what a release request looks like in Slack from start to finish. Some fields have been obscured because they bear no relevance to our setup.

1. The process is initiated either by a Slack user entering a slash-command to bring up the release-request form, or triggered by a Github Action.

2. After this is executed, Slack will post a message in a channel that specifically handles this process. The message will prompt them to Review their request or Cancel.

3. On clicking Review, Slack will contact our API and request the json for the modal screen to create a release request. This will include all the details we need to progress the request and create a ticket in Jira. The image below contains some example text.

4. On clicking submit, the request will be posted to the channel with an @here, to alert all members of the channel so someone can tech review and approve it.

5. Once this has been completed the request will appear in a separate channel where management can perform a final review and approval of the change.

6. The management approver can click on More Details, which will result in Slack hitting our API to request the modal screen json that displays the request’s details.

7. Once management has approved or denied the request it will then send the decision back to the tech approval channel.

8. However, even after approving the request, management can still cancel it later if a scheduling conflict or other reason occurs. They just do this by clicking on the Cancel Release button, as seen below.

9. After clicking the Cancel Release button, it sends a request to our API for a simple one field screen.

10. This cancellation will then update the initial channel with the cancellation and reason.

Automated record-keeping in JIRA

Thanks to our API, the actions being committed in Slack are also being recorded in JIRA without any manual interaction required. I will now illustrate the tickets we create as part of the process.

1. Once the requester has submitted their release request, a ticket in Jira is created with its status kept accurate to its place in the workflow.

2. Once the ticket has been tech approved, the ticket status is updated.

3. Finally, when it has been approved by management, we record it as approved and mark the request status as resolved.

4. If the release is later cancelled, we also update its status and record the reason for cancellation in the ticket.

Adding Github Action functionality

As The Telegraph uses Github, we have added a Github Action that means you can initiate the release request process from a pull request.

By using this method, we can transpose the release description and other fields directly from the pull request into the release request.

We even have another Github Action that we use to template the pull request description, which is also triggered by adding a label to the PR.

This makes life just a little bit easier and ensures consistent data about changes across our platforms.

JIRA calendar that puts changes in context

Finally, we use a calendar-based dashboard in JIRA that is populated using a filter for all the changes we want to track. This gives crucial, at-a-glance context to what changes are happening and when, an important step in approving any new requests that may appear.

The results

Our system has made it simpler for different teams to raise release requests in one place and get them reviewed, even from the comfort of the Slack mobile app. This convenience has made the release process far smoother for us and meant that we can keep pace with the high volume of releases we need to keep maintaining and improving our catalogue of services.

Useful links

Here are some links that will help you if you’re interested in replicating this kind of setup.

• Building an application for Slack
• JIRA (Cloud) Rest API Documents
• Github Actions documentation

Adam Gaudry is a Platforms Engineer at The Telegraph.

Using Slack and JIRA to make releases seamless but safe in a world of microservices was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article looks at how The Telegraph are using the custom badge feature in our recently-integrated, new commenting system, ‘Conversations’ by Viafoura. It’s just one of the tools in their community-building suite, which we chose to implement for its automated moderation and real-time interaction capabilities, to help us drive user engagement. There are also some examples of how to create and assign badges to users by interacting with Viafoura’s API, and how to process a batch of users with a simple Python script.

On all news websites, there exists a never-ending drive to increase user engagement, and having a comments block at the end of an article certainly helps. What’s even better, and makes the comments block more like a conversation, is having your authors contribute to the thread.

Getting an author to stand out against other users in the comment thread is achieved by adding a label or badge next to their name.

In Viafoura Conversations, there are quite a few options you can use for labelling up different types of users, with varying results. ‘Trusted Users’ get an icon next to their name, which looks very similar to the Twitter ‘verified account’ status tick. A ‘Moderator’ user type outputs a similar icon in a different style (‘M’ on a green background), and there is also the capability to set up a basic-looking ‘AUTHOR’ badge by adding some meta tags to the page.

These are all good features, but at The Telegraph we wanted to label our authors with a bespoke ‘Telegraph author’ label, so we needed a more flexible labelling option.

Custom Badges

We found we could achieve this by using Viafoura’s custom badge functionality. It allows you to create badges containing some arbitrary text, and then assign those badges to specific users, so you can label up as many users you like, with anything you like! You can even assign multiple badges to a single user.

Setting up custom badges for your users is not as straightforward as you might think. The functionality is somewhat hidden away and you won’t find it when you look inside Viafoura’s admin console in the User tab, as you might intuitively expect.

In the admin console, the only information you’ll see about a user is their name, email address and a link to the comments they have posted. You might expect to also be able to configure things like badges in here, but sadly (at the time of writing) there is currently no interface for doing this. Instead, you have to make use of Viafoura’s API.

Using Viafoura’s API

To start making calls to Viafoura’s API, you’ll need to know your ‘section_uuid’. This is a unique ID for your domain, provided to you by your client success representative.

Each time you make calls, you’ll need to generate an access token by sending your ‘section_uuid’ to their /authorize_client endpoint. Once you’ve generated your access token, which is valid for 5 minutes at a time, you can then make calls to the api endpoints that you need to create and assign a badge.

First step — Create the badge

To create a badge you need to make a PUT request to the /profile/badge endpoint with a json payload containing the badge label and badge description. The label is the unique identifier and the description is what will actually be displayed on the front-end. For example:

This same call will also update the description in an existing badge if the label matches an existing one. So, if you’ve created a badge by mistake and want to remove it, then just send a DELETE request to the same /profile/badge endpoint with just the label of the badge you want to get rid of in the json payload, for example:

Note that, at the time of writing, there is no endpoint for obtaining a list of all the current badges you have already created in Viafoura’s system, so it’s useful to keep your own record of the badges you’ve created.

Second Step: Assign the Badge

Once you’ve created the badge, you can move on to the next step of assigning it to a specific user. To do this, you’ll need to make a PUT request to the Viafoura API /users/profile/badge endpoint, with a json payload containing user id of the person you want to assign the badge to, their login type, and label of the badge you want to assign to them, for example:

If a 200 response is received back, then you have successfully assigned the badge to the user. If you get a 400 response, then the badge doesn’t exist in Viafoura and will need to be created (see above).

Third Step — Automate!

But what if you have a lot of users that need to be assigned badges? You won’t want to waste time making individual API calls for each person when a simple Python script can do the job.

A Python script to assign badges to a list of users needs to have the following steps:

1. Obtain a list of users to process (which could be stored in a separate file).
2. Make a Viafoura API call to get an Access Token.
3. For each user in the list, make a Viafoura API call to assign the badge.

Note in the following example, the use of the Python requests library which will need to be installed first.

Example Python Script:

That’s it. The last request will return a 200 status code if successful (response.status_code == 200) and your desired badge name will be assigned to the user.

Multiple Users’ Badges

To assign multiple badges to a list of users, you can expand your script and have it read a list of user Ids from a file, loop over the Ids and make the API assignment request.

To get Python to open and read from a plain text file, you can use the Python urllib library. The text file should be set up with each user’s ID on a new line. For example:

After getting the basic Python script working, you can add extras like checking the response codes of your requests, printing out helpful messages and allowing raw inputs for badge names and environments. You could also expand the script to include badge creation and deletion.

For more information and documentation on the badges API, you can refer to Viafoura’s detailed API documentation.

Wendy Cocksedge is a Senior Software Engineer at Telegraph Media Group. Follow her on Twitter.

Viafoura Conversations — Making Custom User Badges using Python was originally published in The Telegraph Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.