<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tim Schill on Medium]]></title>
        <description><![CDATA[Stories by Tim Schill on Medium]]></description>
        <link>https://medium.com/@timschill?source=rss-5e6be4c3a54d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iVruqreEk_4ROZ3UyVmAbw.jpeg</url>
            <title>Stories by Tim Schill on Medium</title>
            <link>https://medium.com/@timschill?source=rss-5e6be4c3a54d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:02:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@timschill/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to use a rerank model in Amazon Bedrock]]></title>
            <link>https://medium.com/@timschill/how-to-use-a-rerank-model-in-amazon-bedrock-5fd74444d0d7?source=rss-5e6be4c3a54d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fd74444d0d7</guid>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[rerank]]></category>
            <category><![CDATA[reranking]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <dc:creator><![CDATA[Tim Schill]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 16:27:20 GMT</pubDate>
            <atom:updated>2024-12-27T16:27:20.374Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to use rerank models in Amazon Bedrock</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BlRdFv4HQKqUbYEb" /></figure><p>During re:Invent 2024 it was <a href="https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-bedrock-rerank-api-accuracy-rag-applications/">announced</a> that Amazon Bedrock now have support for a new kind of model — the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/rerank.html">reranker.</a></p><p>In a typical RAG workflow the user query is converted into vectors through the help of an embedding model, then the vectors are used to carry out a search in a vector database and the result is passed back as context to the prompt and fed to an LLM. In a Proof of Concept the amount of data being searched and returned is often so small that the above workflow works just fine. But in production, the amount of data is probably much bigger and the results may not be as effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J7xrqUWg1eH_Xjen" /></figure><h3>1.Reranker to the rescue</h3><p>This is where a reranker or a rerank model comes into the picture. It acts as a second filter between the vector search and the prompt. Let’s say we want to return up to 50 results from our vector database, but all 50 of these results are most likely not relevant to the user question.</p><p>So we take the 50 results and pass them to our rerank model and tell it we want to get the 5 closest matching results back. The rerank model will evaluate the 50 documents against the user query and give us back 5 results with the highest matching documents based on scoring (scores are calculated between 1 and 0 where better is as close to 1 as possible). We can then take these 5 documents and add them to our prompt as context and carry out the user question with the help of an ordinary LLM. The result should in theory be more precise and accurate.</p><p>Amazon Bedrock currently support two rerank models, Amazon Rerank 1.0 and Cohere Rerank 3.5. Below I will show you how you can work with them through two available methods in Bedrocks API.</p><h3>2.Our Options:</h3><p>The first method is the standard “invoke_model” endpoint in the bedrock-runtime API, and the second option is the “rerank” method available in bedrock-agent-runtime endpoint.</p><p>The bedrock-agent-runtime endpoint is as you probably can guess supposed to be part of an agentic workflow and Amazon Knowledge Bases (another managed feature in Bedrock) but it can be used without having to involve any of these features.</p><p>The reason I will show you both is because I found each of them to have their own strength and weaknesses. For example the “invoke_model” option is more straight and simple. It allows us to pass documents in the form of strings or json. But it wont allow (at least to my knowledge) to involve rank_fields which can be super effective if you are working with json documents. With rank_fields you basically tell the rerank model which keys it should take into consideration while scoring the documents.</p><p>While in the case of the “rerank” method in bedrock-agent-runtime, you get a more complex payload, you can’t mix and match between TEXT (strings) and JSON, but you can leverage the rank_fields.</p><p>So let’s get started, the first approach in this demonstration will be the invoke_model path.</p><pre># Example 1: Using invoke_model with mixed string and JSON documents<br>import boto3<br>import json<br> <br>bedrock_client = boto3.client(&quot;bedrock-runtime&quot;)<br> <br># Prepare the documents, mixing strings and dictionaries<br>documents = [<br>    &quot;Stockholm is the capital of Sweden&quot;,  # String<br>    {&quot;title&quot;: &quot;Bangkok&quot;, &quot;text&quot;: &quot;Bangkok is the capital of Thailand&quot;},  # Dictionary<br>    &quot;Oslo is the capital of Norway&quot;,  # String<br>    &quot;Udon Thani is a city in Thailand&quot;,  # String<br>    &quot;Kuala Lumpur is the capital of Malaysia&quot;,  # String<br>    {&quot;title&quot;: &quot;Vientiane&quot;, &quot;text&quot;: &quot;Vientiane is the capital of Laos&quot;},  # Dictionary<br>]<br> <br># Serialize dictionaries to JSON strings to simulate mixed inputs<br>serialized_documents = [<br>    json.dumps(doc) if isinstance(doc, dict) else doc for doc in documents<br>]<br> <br># Construct the body<br>body = json.dumps(<br>    {<br>        &quot;query&quot;: &quot;What are three cities in south east asia?&quot;,<br>        &quot;documents&quot;: serialized_documents,<br>        &quot;top_n&quot;: 3,<br>        &quot;api_version&quot;: 2,<br>    }<br>)<br> <br># Invoke the Bedrock model<br>response = bedrock_client.invoke_model(<br>    modelId=&quot;cohere.rerank-v3-5:0&quot;,<br>    accept=&quot;application/json&quot;,<br>    contentType=&quot;application/json&quot;,<br>    body=body,<br>)<br> <br># Process the response<br>response_body = json.loads(response.get(&quot;body&quot;).read())<br>print(response_body)<br> <br># Extract the indices of the matching documents<br>matching_indices = [result[&quot;index&quot;] for result in response_body[&quot;results&quot;]]<br> <br># Retrieve and print the matching documents<br>matching_documents = [documents[i] for i in matching_indices]<br> <br>print(&quot;Matching Documents:&quot;)<br>for doc in matching_documents:<br>    # Deserialize JSON strings back to dictionaries for readability<br>    if isinstance(doc, str) and doc.startswith(&quot;{&quot;) and doc.endswith(&quot;}&quot;):<br>        doc = json.loads(doc)<br>    print(doc)</pre><p>Output to the query “What are three cities in south east asia?”</p><pre>{&#39;results&#39;: [<br>  {&#39;index&#39;: 1, &#39;relevance_score&#39;: 0.20046411},<br>  {&#39;index&#39;: 4, &#39;relevance_score&#39;: 0.19896571},<br>  {&#39;index&#39;: 5, &#39;relevance_score&#39;: 0.18133602}<br>]}<br><br>Matching Documents:<br>{&#39;title&#39;: &#39;Bangkok&#39;, &#39;text&#39;: &#39;Bangkok is the capital of Thailand&#39;}<br>Kuala Lumpur is the capital of Malaysia<br>{&#39;title&#39;: &#39;Vientiane&#39;, &#39;text&#39;: &#39;Vientiane is the capital of Laos&#39;}</pre><p>Second approach with “rerank” method in the bedrock-agent-runtime:</p><pre># Example 2: Using rerank with JSON documents and rank_fields<br>import boto3<br>import json<br> <br>bedrock_client = boto3.client(&quot;bedrock-agent-runtime&quot;)<br> <br>payload = {<br>    &quot;queries&quot;: [<br>        {<br>            &quot;textQuery&quot;: {&quot;text&quot;: &quot;What is the capital of Sweden?&quot;},<br>            &quot;type&quot;: &quot;TEXT&quot;,<br>        }<br>    ],<br>    &quot;rerankingConfiguration&quot;: {<br>        &quot;bedrockRerankingConfiguration&quot;: {<br>            &quot;modelConfiguration&quot;: {<br>                &quot;modelArn&quot;: &quot;arn:aws:bedrock:eu-central-1::foundation-model/amazon.rerank-v1:0&quot;,<br>                &quot;additionalModelRequestFields&quot;: {&quot;rank_fields&quot;: [&quot;city&quot;, &quot;text&quot;]},<br>            },<br>            &quot;numberOfResults&quot;: 2,<br>        },<br>        &quot;type&quot;: &quot;BEDROCK_RERANKING_MODEL&quot;,<br>    },<br>    &quot;sources&quot;: [<br>        {<br>            &quot;inlineDocumentSource&quot;: {<br>                &quot;jsonDocument&quot;: {<br>                    &quot;city&quot;: &quot;Stockholm&quot;,<br>                    &quot;text&quot;: &quot;Stockholm is the capital of Sweden.&quot;,<br>                },<br>                &quot;type&quot;: &quot;JSON&quot;,<br>            },<br>            &quot;type&quot;: &quot;INLINE&quot;,<br>        },<br>        {<br>            &quot;inlineDocumentSource&quot;: {<br>                &quot;jsonDocument&quot;: {<br>                    &quot;city&quot;: &quot;Bangkok&quot;,<br>                    &quot;text&quot;: &quot;Bangkok is the capital of Thailand.&quot;,<br>                },<br>                &quot;type&quot;: &quot;JSON&quot;,<br>            },<br>            &quot;type&quot;: &quot;INLINE&quot;,<br>        },<br>        {<br>            &quot;inlineDocumentSource&quot;: {<br>                &quot;jsonDocument&quot;: {<br>                    &quot;city&quot;: &quot;New Stockholm&quot;,<br>                    &quot;text&quot;: &quot;New Stockholm is a city in USA.&quot;,<br>                },<br>                &quot;type&quot;: &quot;JSON&quot;,<br>            },<br>            &quot;type&quot;: &quot;INLINE&quot;,<br>        },<br>    ],<br>}<br> <br># Construct the payload<br>rerank_params = {<br>    &quot;queries&quot;: payload[&quot;queries&quot;],<br>    &quot;rerankingConfiguration&quot;: payload[&quot;rerankingConfiguration&quot;],<br>    &quot;sources&quot;: payload[&quot;sources&quot;],<br>}<br> <br># Invoke the Bedrock model<br>response_body = bedrock_client.rerank(**rerank_params)<br> <br>print(response_body[&quot;results&quot;])<br> <br># Extract the indices of the matching documents<br>matching_indices = [result[&quot;index&quot;] for result in response_body[&quot;results&quot;]]<br> <br># Retrieve and print the matching documents<br>matching_documents = [payload[&quot;sources&quot;][i] for i in matching_indices]<br> <br>print(&quot;Matching Documents:&quot;)<br>for doc in matching_documents:<br>    # Deserialize JSON strings back to dictionaries for readability<br>    if isinstance(doc, str) and doc.startswith(&quot;{&quot;) and doc.endswith(&quot;}&quot;):<br>        doc = json.loads(doc)<br>    print(doc)</pre><p>Output from the query “What is the capital of Sweden?”</p><pre>[<br>  {&#39;index&#39;: 0, &#39;relevanceScore&#39;: 0.981979250907898},<br>  {&#39;index&#39;: 2, &#39;relevanceScore&#39;: 0.006241259165108204}<br>]<br><br>Matching Documents:<br>{&#39;inlineDocumentSource&#39;: {&#39;jsonDocument&#39;: {&#39;city&#39;: &#39;Stockholm&#39;, &#39;text&#39;: &#39;Stockholm is the capital of Sweden.&#39;}, &#39;type&#39;: &#39;JSON&#39;}, &#39;type&#39;: &#39;INLINE&#39;}<br>{&#39;inlineDocumentSource&#39;: {&#39;jsonDocument&#39;: {&#39;city&#39;: &#39;New Stockholm&#39;, &#39;text&#39;: &#39;New Stockholm is a city in USA.&#39;}, &#39;type&#39;: &#39;JSON&#39;}, &#39;type&#39;: &#39;INLINE&#39;}</pre><p>I must say that I do favour the bedrock-agent-runtime approach because it can leverage the “rank_fields” but it do add some complexity to the payload being sent to bedrock.</p><h3>Conclusion</h3><p>Overall I am really happy to see Amazon Bedrock implement support for not one but two great rerank models. I know this is a feature many people have been waiting for. And I hope people start to implement these kind of models into their RAG workflow because they can make a big difference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fd74444d0d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Chatbots with Amazon Bedrock & LangChain]]></title>
            <link>https://medium.com/aws-specialists/how-to-build-chatbots-with-amazon-bedrock-langchain-a37a6196cbb1?source=rss-5e6be4c3a54d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a37a6196cbb1</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Tim Schill]]></dc:creator>
            <pubDate>Fri, 29 Sep 2023 14:33:21 GMT</pubDate>
            <atom:updated>2023-10-27T06:47:20.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*crDKcKzHcf5d8tNEvUg9HQ@2x.png" /></figure><p><a href="https://aws.amazon.com/bedrock/">Amazon Bedrock</a> was released on the 28th of September, and I have been fortunate enough to have had access to it for some time now while it has been in closed preview.</p><p><strong>If you are reading this article, I’m sure you already know about Amazon Bedrock; if not, let’s summarize it quickly.</strong> Amazon Bedrock is a fully managed service that gives you access to foundation models (FMs). Being serverless means there is no infrastructure to care about. However, you still have access to some powerful features (for some models) like fine-tuning with your own data and agents, which you can look at as tools you can create yourself to supercharge your FM with abilities it usually doesn’t have. For example, you could create an agent to query a database or talk with an external API.</p><p>You also can create context-aware applications with the help of Related Augmented Generation (RAG). We will not dive deeper than that in this article.</p><h3>The Setup</h3><p>Getting started with Bedrock is simple, but to utilize its full power, you can add<a href="https://www.langchain.com/"> LangChain</a> to the mix. Langchain is a framework that will simplify the creation of applications using large language models (LLMs). Its power lies in its ability to “chain” or combine multiple components.</p><p>In this example, we will create a chatbot with the help of Streamlit, LangChain, and its classes “ConversationChain” and “ConversationBufferMemory.”</p><p>First, we create a new Python file. Let’s call it “bedrock.py”.</p><pre>import os<br>import boto3<br>from langchain.chains import ConversationChain<br>from langchain.llms.bedrock import Bedrock<br>from langchain.memory import ConversationBufferMemory<br>from langchain.prompts import PromptTemplate</pre><p>We import some libraries from LangChain. Let’s quickly go through them. Bedrock will allow us to create an object with details about what FM we want to use and configure model parameters, authentication, etc. PromptTemplate will enable us to create prompts we can ingest variables into, similar to Python f-strings.</p><p>ConversationBufferMemory allows us to take control of the history or memory efficiently. And finally, ConversationChain will tie all these objects together in a chain.</p><p>Next we will create the chain function.</p><pre>def bedrock_chain():<br>    profile = os.environ[&quot;AWS_PROFILE&quot;]<br><br>    bedrock_runtime = boto3.client(<br>        service_name=&quot;bedrock-runtime&quot;,<br>        region_name=&quot;us-east-1&quot;,<br>    )<br><br>    titan_llm = Bedrock(<br>        model_id=&quot;amazon.titan-text-express-v1&quot;, client=bedrock_runtime, credentials_profile_name=profile<br>    )<br>    titan_llm.model_kwargs = {&quot;temperature&quot;: 0.5, &quot;maxTokenCount&quot;: 700}<br><br>    prompt_template = &quot;&quot;&quot;System: The following is a friendly conversation between a knowledgeable helpful assistant and a customer.<br>    The assistant is talkative and provides lots of specific details from it&#39;s context.<br><br>    Current conversation:<br>    {history}<br><br>    User: {input}<br>    Bot:&quot;&quot;&quot;<br>    PROMPT = PromptTemplate(<br>        input_variables=[&quot;history&quot;, &quot;input&quot;], template=prompt_template<br>    )<br><br>    memory = ConversationBufferMemory(human_prefix=&quot;User&quot;, ai_prefix=&quot;Bot&quot;)<br>    conversation = ConversationChain(<br>        prompt=PROMPT,<br>        llm=titan_llm,<br>        verbose=True,<br>        memory=memory,<br>    )<br><br>    return conversation</pre><p>Our “bedrock_chain” function will create our Bedrock object. In our case, we use the Titan Text G1 — Express model. We then make our prompt template. Our prompt holds three essential parts: the instruction, the context (history), and our user query (input). We then configure the memory attribute with the help of “ConversationBufferMemory.” And finally, we put all this together by creating a “ConversationChain” object.</p><p>And we will end with creating two functions.</p><pre>def run_chain(chain, prompt):<br>    num_tokens = chain.llm.get_num_tokens(prompt)<br>    return chain({&quot;input&quot;: prompt}), num_tokens<br><br><br>def clear_memory(chain):<br>    return chain.memory.clear()</pre><p>The first one, run_chain will be used when we call our chain from our Streamlit app. And clear_memory is pretty obvious; it will empty our history.</p><p>We now have everything we need to communicate with Amazon Bedrock. In the next step, we will create our <a href="https://streamlit.io/">Streamlit</a> app.</p><p>We start with creating a new file called “app.py” and import a couple of libraries.</p><pre>import streamlit as st<br>import uuid<br>import bedrock</pre><p>There is nothing strange here. Streamlit is needed, uuid is used to handle user sessions, and Bedrock is the file we created before.</p><p>Now, let’s configure our session_state in Streamlit.</p><pre>USER_ICON = &quot;images/user-icon.png&quot;<br>AI_ICON = &quot;images/ai-icon.png&quot;<br><br>if &quot;user_id&quot; in st.session_state:<br>    user_id = st.session_state[&quot;user_id&quot;]<br>else:<br>    user_id = str(uuid.uuid4())<br>    st.session_state[&quot;user_id&quot;] = user_id<br><br>if &quot;llm_chain&quot; not in st.session_state:<br>    st.session_state[&quot;llm_app&quot;] = bedrock<br>    st.session_state[&quot;llm_chain&quot;] = bedrock.bedrock_chain()<br><br>if &quot;questions&quot; not in st.session_state:<br>    st.session_state.questions = []<br><br>if &quot;answers&quot; not in st.session_state:<br>    st.session_state.answers = []<br><br>if &quot;input&quot; not in st.session_state:<br>    st.session_state.input = &quot;&quot;</pre><p>Next, we create a function to create our top bar, add a button for clearing the chat, and an if-statement with some clearing functionality.</p><pre>def write_top_bar():<br>    col1, col2, col3 = st.columns([2, 10, 3])<br>    with col2:<br>        header = &quot;Amazon Bedrock Chatbot&quot;<br>        st.write(f&quot;&lt;h3 class=&#39;main-header&#39;&gt;{header}&lt;/h3&gt;&quot;, unsafe_allow_html=True)<br>    with col3:<br>        clear = st.button(&quot;Clear Chat&quot;)<br><br>    return clear<br><br><br>clear = write_top_bar()<br><br>if clear:<br>    st.session_state.questions = []<br>    st.session_state.answers = []<br>    st.session_state.input = &quot;&quot;<br>    bedrock.clear_memory(st.session_state[&quot;llm_chain&quot;])</pre><p>We can now create the main function for handling input from the user.</p><pre>def handle_input():<br>    input = st.session_state.input<br><br>    llm_chain = st.session_state[&quot;llm_chain&quot;]<br>    chain = st.session_state[&quot;llm_app&quot;]<br>    result, amount_of_tokens = chain.run_chain(llm_chain, input)<br>    question_with_id = {<br>        &quot;question&quot;: input,<br>        &quot;id&quot;: len(st.session_state.questions),<br>        &quot;tokens&quot;: amount_of_tokens,<br>    }<br>    st.session_state.questions.append(question_with_id)<br><br>    st.session_state.answers.append(<br>        {&quot;answer&quot;: result, &quot;id&quot;: len(st.session_state.questions)}<br>    )<br>    st.session_state.input = &quot;&quot;</pre><p>The important part is lines 6–8. This is where we initialize our chain, call it, and store the result. Next is to create a couple of functions to render the question, answer, and our history.</p><p>And finally, we call all the functions and add our input form.</p><pre>def write_user_message(md):<br>    col1, col2 = st.columns([1, 12])<br><br>    with col1:<br>        st.image(USER_ICON, use_column_width=&quot;always&quot;)<br>    with col2:<br>        st.warning(md[&quot;question&quot;])<br>        st.write(f&quot;Tokens used: {md[&#39;tokens&#39;]}&quot;)<br><br><br>def render_answer(answer):<br>    col1, col2 = st.columns([1, 12])<br>    with col1:<br>        st.image(AI_ICON, use_column_width=&quot;always&quot;)<br>    with col2:<br>        st.info(answer[&quot;response&quot;])<br><br><br>def write_chat_message(md):<br>    chat = st.container()<br>    with chat:<br>        render_answer(md[&quot;answer&quot;])<br><br><br>with st.container():<br>    for q, a in zip(st.session_state.questions, st.session_state.answers):<br>        write_user_message(q)<br>        write_chat_message(a)<br><br><br>st.markdown(&quot;---&quot;)<br>input = st.text_input(<br>    &quot;You are talking to an AI, ask any question.&quot;, key=&quot;input&quot;, on_change=handle_input<br>)</pre><p>That’s it; we can now start our application by typing “streamlit run app.py” and start chatting.</p><p>Our chatbot built with Streamlit, LangChain and Amazon Bedrock</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*ax3Re-fmKkJ7p0LVQS6a0w.png" /><figcaption>Our chatbot built with Streamlit, LangChain and Amazon Bedrock</figcaption></figure><h3>What’s next?</h3><p>I recommend you look deeper at LangChain if you are not already familiar with it. You can also look at the <a href="https://github.com/aws-samples">aws-samples</a> Github page; they have some great examples to get you started. For example, you could add <a href="https://aws.amazon.com/kendra/">Amazon Kendra</a> to the mix. Connect it with one of its many sources, like Atlassian Confluence, and set up Langchain to utilize the Kendra retriever. And now you have a chatbot that can answer questions based on the context it grabs from your internal confluence wiki pages.</p><h3>In Conclusion</h3><p>Amazon Bedrock is super easy to start with; it does not require many lines of code, and with the help of LangChain, you can create some really powerful applications. And if you add agents to the mix, the possibilities are almost limitless.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a37a6196cbb1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aws-specialists/how-to-build-chatbots-with-amazon-bedrock-langchain-a37a6196cbb1">How to Build Chatbots with Amazon Bedrock &amp; LangChain</a> was originally published in <a href="https://medium.com/aws-specialists">AWS Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon CloudWatch: Unveiling and Managing Stealthy Log Costs]]></title>
            <link>https://medium.com/aws-specialists/amazon-cloudwatch-unveiling-and-managing-stealthy-log-costs-bf3ca97ff21a?source=rss-5e6be4c3a54d------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf3ca97ff21a</guid>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[cloudwatch-logs]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Tim Schill]]></dc:creator>
            <pubDate>Thu, 15 Jun 2023 10:48:21 GMT</pubDate>
            <atom:updated>2023-06-15T10:50:32.153Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ls7p0Jy-B_3t1yztZmFQDw.png" /></figure><p><a href="https://aws.amazon.com/cloudwatch/">Amazon CloudWatch</a>, AWS’s omnipotent monitoring service, is invaluable in any AWS specialist’s toolkit. It offers us a vast array of metrics, alarms, dashboards, and insights that can help keep an eagle eye on AWS resources. But with great power comes a responsibility — managing the cost. The financial implications of CloudWatch, particularly ingestion, can be unexpectedly significant if not properly addressed.</p><p>This blog post will dissect the components contributing to stealthy CloudWatch costs and offer tactics to optimize them without compromising your monitoring capabilities.</p><h3>CloudWatch Log Storage: The Breakdown</h3><p>AWS operates on a pay-as-you-go philosophy; you pay for what you use. CloudWatch Log pricing primarily depends on data ingestion, storage, and analysis.</p><p>Ingestion and storage charges often become the culprits behind unexpected CloudWatch expenses. You are billed $0.50 per GB ingested and $0.03 per GB-Mo. As you can see, the cost of ingesting data is much more expensive than the cost of storage. So, we want to ensure we only send the data we need to CloudWatch.</p><h3>Strategies to Curb CloudWatch Log Storage Costs</h3><h4>1. Streamline Your Logging</h4><p>One of the crucial steps to managing your CloudWatch log storage costs lies in streamlining your logging. Unfortunately, an all-too-common oversight is excessive logging, which often passes under the radar but significantly bloats your storage costs.</p><p>Consider this: do all your logs contribute valuable insights? Redundant and verbose logging may seem harmless, but it can rapidly inflate your storage costs while offering minimal added value. Therefore, conducting a comprehensive review of your logging practices and streamlining them to maintain only logs that deliver substantial insights is crucial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWHtdUdvr0YOonT3Dwvscw.jpeg" /><figcaption>Search for log levels in the CloudWatch Logs console.</figcaption></figure><p>Identify the components that create noise in your logs and fine-tune their logging levels to optimize their output. Take <a href="https://solr.apache.org/">Apache Solr</a> as a case in point. Enabling <a href="https://sematext.com/blog/logging-levels/">INFO logging</a> here can provide detailed search statistics, which is an asset when troubleshooting search performance issues. But beware! It can also lead to an unexpected cost surge into the thousands every month, especially if your Apache Solr server experiences heavy traffic.</p><p>A savvy approach would be to use INFO logging sparingly and only activate it when needed for troubleshooting. For example, in normal operation, you should limit your Apache Solr messages to WARN and ERROR levels, which can be directly sent to CloudWatch. This strategy can offer significant cost benefits without compromising on the required functionality.</p><h4>2. Leverage Log Expiry</h4><p>CloudWatch Logs lets you set an expiry period for your logs. After this period, logs get automatically deleted, saving storage. By default, logs are kept indefinitely. Review your retention policies and align them with your compliance and business needs. Great tools, such as <a href="https://github.com/binxio/aws-cloudwatch-log-minder">aws-cloudwatch-log-minder</a>, can help you automate this process.</p><figure><img alt="Retention period" src="https://cdn-images-1.medium.com/max/1024/1*xiwUqsvBLyltb4FQkaxFkQ.png" /><figcaption>The predefined retention period in the CloudWatch Logs console.</figcaption></figure><h4>3. Optimize Data Ingestion</h4><p>Data ingestion charges in CloudWatch, incurred when you transmit logs, can accumulate incrementally if not adeptly managed. Therefore, before sending logs to CloudWatch, optimizing data ingestion by implementing log filters and patterns is crucial. This approach ensures that only pertinent logs, indispensable for your operations, are transmitted.</p><p>This optimization can be achieved in several ways, depending on the resources at your disposal. Some applications, such as <a href="https://httpd.apache.org/">Apache</a>, can provide support for this task. For example, Apache’s error.log and access.log features can be employed to fine-tune what logs get sent to CloudWatch. In other scenarios, the configuration can be made directly in the logging framework, as seen in log4j in Java or logger in Python. However, in cases where these detailed configurations are not supported, setting the appropriate log level remains a feasible alternative.</p><figure><img alt="Log levels" src="https://cdn-images-1.medium.com/max/1024/1*92_JA-nTV1fuD8MVFEI-dA.jpeg" /><figcaption>Log levels explained in text and numeric format.</figcaption></figure><p>A clever approach is segregating log levels into different files within your application. You could, for example, channel WARN and ERROR messages into a separate file and directly ingest that into CloudWatch. INFO messages, on the other hand, could be stored in a different file where log rotation can be enabled. Keeping them locally on the instance and utilizing batch uploads to S3 can offer a cost-effective solution, especially if you need to maintain them beyond instance termination.</p><figure><img alt="Handle different log levels in Python" src="https://cdn-images-1.medium.com/max/1024/1*43mBt10ejRUv1OZDm0yaEw.png" /><figcaption>Handle different log levels in Python.</figcaption></figure><p>If the separation of log levels proves challenging, carefully evaluate the utility of the information you consider for ingestion. In some cases, WARN and ERROR levels may be sufficient. For applications where you have control over the logged information, ensure you include only valuable, actionable details in your log messages.</p><figure><img alt="Only send NGINX error logs to CloudWatch" src="https://cdn-images-1.medium.com/max/1024/1*Ry9qWIQuxStN-PWbJibdOA.png" /><figcaption>Configure the CloudWatch Logs agent to only import NGINX error logs.</figcaption></figure><p>It’s worth noting that certain AWS services like <a href="https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html">VPC flow logs</a>, <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html">ELB access logs</a>, and <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html">CloudFront</a> access logs offer an alternative: storing logs directly on S3 instead of CloudWatch. When coupled with using <a href="https://aws.amazon.com/athena/">Athena</a> for searching through logs as needed, this option can result in considerable cost savings. For these logs, opting for S3 storage can halve your costs. As always, strategic and thoughtful management of log storage and data ingestion can go a long way in optimizing your CloudWatch expenditure.</p><h4>4. Be wary of automated CloudWatch Logs Insights queries.</h4><p>We have seen examples where developers have created automated queries in tools such as Grafana to query for log data they present in graphs. This can quickly rack up charges in hundreds of dollars every day if these queries scan through a lot of data. A better way is to create a metric filter on the log group. You can present this as a graph in a Dashboard and connect alarms to it.</p><figure><img alt="CloudWatch Metric Filters" src="https://cdn-images-1.medium.com/max/798/1*rLfQ0MmGsW1XwMlvY7Slyw.png" /><figcaption>Create metric filters through the CloudWatch Logs console.</figcaption></figure><h3>Conclusion</h3><p>While CloudWatch is a mighty service, it’s essential for us as AWS specialists to keep a watchful eye on the costs associated with it. By optimizing our logging practices, leveraging built-in features, and thinking strategically about what and how we’re logging, we can prevent our CloudWatch log storage costs from spiraling out of control.</p><p>Remember, cost optimization in AWS is an ongoing process — regular audits and revisions of your usage are keys to cost-effective cloud management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf3ca97ff21a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aws-specialists/amazon-cloudwatch-unveiling-and-managing-stealthy-log-costs-bf3ca97ff21a">Amazon CloudWatch: Unveiling and Managing Stealthy Log Costs</a> was originally published in <a href="https://medium.com/aws-specialists">AWS Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Six Ways to Cut Costs with EC2 | AWS Specialists]]></title>
            <link>https://medium.com/aws-specialists/six-ways-to-cut-costs-with-ec2-858f0d27d366?source=rss-5e6be4c3a54d------2</link>
            <guid isPermaLink="false">https://medium.com/p/858f0d27d366</guid>
            <category><![CDATA[rightsizing]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-graviton]]></category>
            <category><![CDATA[aws-cost-optimization]]></category>
            <dc:creator><![CDATA[Tim Schill]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 15:34:40 GMT</pubDate>
            <atom:updated>2023-04-13T13:41:39.158Z</atom:updated>
            <content:encoded><![CDATA[<h3>Six Ways to Cut Costs with EC2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6fTv4xFRewgywXzWTC-4nA@2x.png" /></figure><p>AWS EC2, or Amazon Elastic Compute Cloud, is incredibly powerful and one of the most widely used cloud computing services. The right optimization can help you cut costs without sacrificing performance.</p><p>Here are six ways you can cut costs with EC2.</p><h3>1. Rightsizing</h3><p>The first step to cost optimization is ensuring you are running the right instance type for your workload. Rightsizing involves choosing the right instance type and the right instance size to meet the performance requirements of your workload. <strong>When done right and all your workloads run on optimized instances that aren’t wasting resources, you can likely cut costs by around 20–30%.</strong></p><p>AWS provides excellent tools to help with rightsizing, such as <a href="https://aws.amazon.com/compute-optimizer/">AWS Compute Optimizer</a> or <a href="https://aws.amazon.com/premiumsupport/technology/trusted-advisor/">Trusted Advisor</a>. We also want to remind you of the importance of resizing your instances with the latest version of the instance type. For example, if you are currently running a bunch of m5.2xlarge instances, see if you can convert these into m6i.2xlarge instances instead. The m6i instance family is priced the same as the m5 family, but it will give you better CPU performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVEUA9bAUCZexKq4NHtOUA.png" /><figcaption>AWS Compute Optimizer</figcaption></figure><h3>2. Closely monitor scheduled instances</h3><p>Another cost optimization strategy is to schedule instances to turn on and off when not in use. This is especially useful for development and testing environments, which are only needed during working hours. In scenarios where there are predictable scheduled intervals for running compute services, <strong>using </strong><a href="https://aws.amazon.com/solutions/implementations/instance-scheduler-on-aws/"><strong>AWS Instance Scheduler</strong></a><strong> can generate savings of up to 70%.</strong></p><h3>3. Spot instances</h3><p><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html">Spot instances</a> are cost-effective for batch workloads, big data and analytics, and other stateless workloads. <strong>Spot instances can save you up to 90% of the on-demand price</strong>, but know that AWS can take back the instance at any time with a 2-minute notice, making the option best suited for stateless workloads. One of the best things about this? You can use spot instances and on-demand instances in conjunction with each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8yztI6s6kdqFZqKUH9pAA.png" /><figcaption>Spot instance pricing history</figcaption></figure><h3>4. Optimizing Auto Scaling Groups (ASG)</h3><p>Another way to optimize costs is by using <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html">auto scaling groups</a>, or ASG for short. ASG can automatically adjust the number of instances in your fleet based on demand. You can also configure ASG to use spot instances when needed, <strong>resulting in further cost savings.</strong> There is no additional charge for AWS Auto Scaling. You only pay for the AWS resources needed to run your applications and Amazon CloudWatch monitoring fees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nE0Gf3yrp-Ej9-CDIwNzmw.png" /><figcaption>Instances in our auto scaling group</figcaption></figure><h3>5. Reserved instances &amp; Compute Savings Plans</h3><p>One of the ways to cut costs with EC2 involves using reserved instances or compute savings plans. Reserved instances <strong>give you a discount of up to 72%</strong> on your instances and provide a more predictable cost structure. Compute savings plans offer a similar discount (up to 62%) but with more flexibility, covering not only EC2 instances but also Lambda and Fargate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rc76QwfDowoftb3DaYEW2Q.png" /><figcaption>Purchase reserved instances</figcaption></figure><h3>6. Amazon Graviton</h3><p>Our sixth and final recommendation for ways to cut costs with EC2 involves using <a href="https://aws.amazon.com/ec2/graviton/">Amazon Graviton</a> and the ARM architecture, which is generally less expensive than x86 processors and optimized for cloud workloads, and can be easily scaled as needed through AWS.</p><p>Leaping to Graviton can be a bigger task since you need to do more than just convert the instances. In order to be able to run Graviton, you have to compile your application for the ARM64 architecture. If you also rely on third-party dependencies in your application, you must ensure that these also support ARM64.</p><p>Once you have converted your workloads to ARM64, <strong>you can expect up to 40% better price performance</strong> than the Intel-based instances. Meaning you will not only get a lower price, but you will also receive better CPU performance out of the same amount of cores in your instances. You can use Graviton for your RDS and Elasticache instances as well if you meet the requirements, typically being on the supported version of the database engine. We recommend running Graviton wherever possible for performance and cost savings.</p><h3>Wrap-up</h3><p>In addition to these six strategies, there are several other ways to optimize costs with EC2, such as optimizing EBS volume size and IOPS reservations, switching from GP2 to GP3 volumes, cleaning up unused AMI and snapshots, and removing unused Elastic IPs.</p><p>Now, what are your ways to cut EC2 costs? I would love to hear about it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=858f0d27d366" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aws-specialists/six-ways-to-cut-costs-with-ec2-858f0d27d366">Six Ways to Cut Costs with EC2 | AWS Specialists</a> was originally published in <a href="https://medium.com/aws-specialists">AWS Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set up your AWS Budgets like the Professionals]]></title>
            <link>https://medium.com/aws-specialists/set-up-your-aws-budgets-like-the-professionals-473cf2728ac4?source=rss-5e6be4c3a54d------2</link>
            <guid isPermaLink="false">https://medium.com/p/473cf2728ac4</guid>
            <category><![CDATA[aws-budgets]]></category>
            <category><![CDATA[aws-cost-optimization]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-cost]]></category>
            <category><![CDATA[cost-control]]></category>
            <dc:creator><![CDATA[Tim Schill]]></dc:creator>
            <pubDate>Tue, 14 Mar 2023 12:22:04 GMT</pubDate>
            <atom:updated>2023-03-14T21:57:55.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXzTvrNd8hnDM8HAsDduXA@2x.png" /></figure><p>The cost of running your applications and services on AWS can quickly spiral out of control if you are not careful. One of the most important things you can do to prevent this is to utilize the <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/">AWS Budgets service</a>. If you do it like the professionals, it will improve planning, and cost control, meanwhile providing forecasts and helping you take action if costs get out of hand.</p><h3>Setting up budgets in AWS</h3><p>Setting up budgets in AWS is very simple — you can create a budget from the AWS Billing and Cost Management dashboard. To create a budget, you must specify the amount you want to spend, the time period (such as monthly), and what you want to monitor. For example, you can set up a budget for a specific service, usage types or usage type groups, savings plans, reservations, or the overall cost.</p><figure><img alt="The Overview of the AWS Budget console" src="https://cdn-images-1.medium.com/max/1024/0*a2d4bALwJ1BChgAr" /><figcaption>The overview of the AWS budget console.</figcaption></figure><p><strong>Whenever you are on the AWS free tier,</strong> spinning up instances, launching databases, and setting up load balancers, you want to ensure you don’t exceed the free tier. Therefore, creating a budget notifying you once you have accumulated an actual cost of $0.01<strong> </strong>could be a great way to help you terminate that risk.</p><figure><img alt="An AWS budget that will notify if you’re exceeding your free tier quota" src="https://cdn-images-1.medium.com/max/1024/0*khw9v2Gc0eqvjFC8" /><figcaption><em>Terminate the risk of exceeding the free tier quota with AWS Budgets</em></figcaption></figure><h3>Keep development costs in check</h3><p>If you are part of or helping an organization with many developers working on different accounts, you might need help keeping track of all the resources being used. This can lead to a situation where some resources are no longer required but are still being charged for, resulting in unnecessary costs. To avoid this, you can set up AWS budgets for each dev account.</p><p>Once the budget is set, you can also set up an alert to notify the team leader or person in charge of that account when the budget is getting close to being exceeded. This helps the team to be aware of their spending and take action if needed.</p><p>By delegating the responsibility of cleaning up resources to each team with knowledge of their usage, you can avoid the need for a dedicated team to handle this task for the whole organization. This helps in reducing costs and streamlining the DevOps process.</p><h3>The 80% rule</h3><p>If you exceed your budget, you can specify what action you want AWS to take, such as sending you an email alert or denying new instances from running. By setting up alerts to notify you when you have spent a certain percentage of your budget, such as 80%, you will receive early warnings of rising costs. This will give you time to take action to prevent overspending. Taking care of minor issues before they become full-scale problems is incredibly important.</p><p>With <a href="https://medium.com/aws-specialists/how-to-use-the-cost-allocation-tags-in-aws-445be6bf19a8">Cost Allocation Tags</a>, you can also use the AWS Budgets service to set budget alerts for specific tags to notify you if costs exceed a certain threshold.</p><h3>Master your finances with Forecasts</h3><p>To make informed decisions on AWS infrastructure, you should monitor the forecasting provided by AWS and get insights into expected usage and costs. You can set it up as an alarm metric for forecasted costs (just as for actual costs) to receive notifications well in advance if changes to the infrastructure are at risk of exceeding your budget. With forecasting, you can plan for future growth and investment in AWS resources, allowing you to make strategic decisions about expanding your infrastructure, scaling down, or introducing new services.</p><p>However, it’s essential to be aware that the forecasts may not always be accurate and that you can’t interfere with how the forecast is done; only specify for what resources you want to be forecasted.</p><h3>Wrap-up</h3><p>Setting up AWS Budgets is crucial to prevent unexpected costs and gain control over your AWS spending. Creating budgets is easy and allows you to monitor and control costs, set up alerts, and utilize forecasting tools. By following AWS Specialist best practices, you can optimize usage, manage costs effectively, and make informed decisions about your AWS infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=473cf2728ac4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aws-specialists/set-up-your-aws-budgets-like-the-professionals-473cf2728ac4">Set up your AWS Budgets like the Professionals</a> was originally published in <a href="https://medium.com/aws-specialists">AWS Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use the Cost Allocation Tags in AWS]]></title>
            <link>https://medium.com/aws-specialists/how-to-use-the-cost-allocation-tags-in-aws-445be6bf19a8?source=rss-5e6be4c3a54d------2</link>
            <guid isPermaLink="false">https://medium.com/p/445be6bf19a8</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-cost-optimization]]></category>
            <category><![CDATA[cost-allocation-tags]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[Tim Schill]]></dc:creator>
            <pubDate>Thu, 02 Feb 2023 18:30:48 GMT</pubDate>
            <atom:updated>2023-02-02T18:47:24.197Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="How to use Cost Allocation Tags in AWS" src="https://cdn-images-1.medium.com/max/1024/1*LDOu1ugtNpy1gy8-fusXjQ@2x.png" /></figure><p>Cost allocation tags are a powerful tool that can help organizations track and allocate costs associated with their AWS resources. These tags can identify specific resources, such as development or production environments, and assign costs to the appropriate department or project. This article will cover best practices and how to avoid pitfalls when using cost allocation tags.</p><h3>Design your tagging strategy</h3><p>The first step when designing your tagging strategy is to identify the areas you want to be able to filter on. For example, it could be departments or teams, applications, or you may want to split the application into storage and compute. It can also be regions, environments (prod dev stage), or even specific people who own the resources.</p><figure><img alt="Use your tagging strategy on resources" src="https://cdn-images-1.medium.com/max/1024/0*TuBimxhNT1bCnLSQ" /></figure><p>When designing your tagging strategy, it is crucial to establish a consistent and easy-to-understand naming convention for your tags, e.g., <em>environment = prod, application = webserver, team = frontend, owner = john_doe</em>. Refrain from mixing and matching upper and lowercase letters. Please keep it simple, lowercase everywhere. Also, try to avoid having whitespace in tags; this can cause trouble in some AWS services console experiences. It is also preferred to programmatically tag resources without human interaction to ensure they are consistently named.</p><h3>The “billing” tag</h3><p>Let’s say your application consists of multiple resources, and you install this application numerous times for different customers. The tag <em>application</em> would then provide the combined cost for all the application installations. Of course, you could combine multiple tags, but we prefer to have a separate tag we call billing that takes the values of other tags and combines them into one.</p><p>For example, suppose we want to use the tags <em>application = frontend, customer = evolate, and environment = dev</em> to give us the total cost for this particular installation in our dev environment for the customer evolate. In that case, our billing tag could look like this <em>billing = evolate_frontend_dev</em>. Having the billing tag will enable us to group on it in Cost Explorer and give us a nice separation of cost for our different applications as a whole.</p><h3>Implementation of tags</h3><p>Once your tags have been designed, the next step is to implement them. This can be done by creating tags at the time of resource creation or by applying them to existing resources. If you use Terraform, providing the tags at the provider level is a great option. Set tags at the resource level where extra detail is needed. For example, <em>application_type</em> could be a resource tag not provided at the provider level but rather at the resource level.</p><p>It is also highly recommended that you include steps in your pipelines to verify that tags have been provided. If not, they should fail. In the case of Terraform, you can use a tool such as <a href="https://github.com/aquasecurity/tfsec">tfsec</a> to enforce tags in your templates.</p><p>You may already have resources in AWS that either need more tags or maybe have no tags at all. This is because they might have been set up manually through the console. Then you can use the “Tag Editor” found under <a href="https://aws.amazon.com/resourceexplorer/">AWS Resource Explorer</a> to set tags on multiple resources faster than going over them manually.</p><figure><img alt="Find resources to tag in the AWS Tag editor" src="https://cdn-images-1.medium.com/max/1024/0*kkU0SQ91Sk4G5j6I" /></figure><p>AWS Organizations also allows you to set up <a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_tag-policies.html">tag policies</a>. A tag policy specifies a set of tags and the allowed values for these tags, which I recommend you set up after implementing your tags.</p><h3>Enable cost allocation tags</h3><p>When you feel comfortable and know what tags you need for your strategy, it is time to enable them as cost allocation tags. You do this under your billing dashboard, and on the left menu, find “Cost allocation tags.” Locate the tags you want to enable, and be aware that the tags must exist already before you can find them. Enable the checkbox for the tags and click enable. It can take up to 24 hours before the tags have been activated, and you can start to use them.</p><figure><img alt="Enable cost allocation tags in billing console" src="https://cdn-images-1.medium.com/max/1024/0*0sX0A-gA3XHKoWdc" /></figure><h3>Monitoring costs and maintenance of tags</h3><p>Once your tags have been implemented, the next step is to monitor and analyze your costs. This can be done using the <a href="https://aws.amazon.com/aws-cost-management/aws-cost-explorer/">AWS Cost Explorer</a>, which allows you to view and analyze your costs by tag. You can also use the <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/">AWS Budgets</a> service to set budget alerts for specific tags to notify you if costs exceed a certain threshold.</p><figure><img alt="Cost explorer" src="https://cdn-images-1.medium.com/max/1024/0*R567cy5yu1KTCjJ-" /></figure><p>Finally, it is essential to review and update your tags regularly. This can be done by reviewing the tag policies you created. It will tell you what resources that comply with your tag policies. It is also essential to check your tagging strategy and make any necessary changes to ensure that your tags are accurate and effective.</p><figure><img alt="AWS Organization Tag Policies" src="https://cdn-images-1.medium.com/max/1024/0*qrStuE9OQ7MY6x96" /></figure><h3>But wait, Cost Categories?</h3><p>AWS released <a href="https://aws.amazon.com/aws-cost-management/aws-cost-categories/">Cost Categories</a> in November 2019, and it can help you to map your cost and usage information in a more advanced structure than Cost Explorer can do on its own.</p><p>For example, you can set up rules for dimensions, such as tags and accounts, and group cost on identifiers in account names, charge types, services, regions, etc. We will follow up on Cost Categories in its own post, so make sure to follow us, so you don’t miss out!</p><h3>TL;DR</h3><p>Cost allocation tags in AWS can help track and allocate resource costs. When designing your tagging strategy, establish a consistent naming convention and tag resources programmatically. Implement tags through resource creation or by applying them to existing resources. Monitor and analyze costs using the AWS Cost Explorer and regularly update tags for accuracy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=445be6bf19a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aws-specialists/how-to-use-the-cost-allocation-tags-in-aws-445be6bf19a8">How to use the Cost Allocation Tags in AWS</a> was originally published in <a href="https://medium.com/aws-specialists">AWS Specialists</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>