<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yuki Nagae on Medium]]></title>
        <description><![CDATA[Stories by Yuki Nagae on Medium]]></description>
        <link>https://medium.com/@yukinagae?source=rss-63b3f8a4e2d5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fv35DLVE-PClGbsn416pvQ.png</url>
            <title>Stories by Yuki Nagae on Medium</title>
            <link>https://medium.com/@yukinagae?source=rss-63b3f8a4e2d5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:35:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yukinagae/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Firebase Genkit for Go Developers: A Guide to Building LLM Applications]]></title>
            <link>https://medium.com/firebase-developers/firebase-genkit-for-go-developers-a-guide-to-building-llm-applications-f96c51c34b10?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f96c51c34b10</guid>
            <category><![CDATA[ai-orchestration]]></category>
            <category><![CDATA[firebase]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genkit]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Fri, 25 Oct 2024 12:43:55 GMT</pubDate>
            <atom:updated>2024-11-20T12:50:41.331Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKDa1ZG8ZJl8XNs_AkAixA.png" /></figure><h3>Introduction</h3><p>When developing LLM applications, JavaScript/TypeScript and Python are often the primary programming language choices. However, it’s now possible to develop LLM applications using Go as well. This article will introduce how to do that. As a simple example, we’ll create an API that summarizes web content when given a URL.</p><p>The final product is equivalent to the TypeScript implementation below, so reading it will deepen your understanding of Firebase Genkit:</p><p><a href="https://medium.com/@yukinagae/your-first-guide-to-getting-started-with-firebase-genkit-6948d88e8a92">Your First Guide to Getting Started with Firebase Genkit</a></p><p>Note: As Genkit’s Go support is in alpha, specifications may change significantly or contain bugs. It is recommended for private or experimental purposes only!</p><h3>What is Firebase Genkit?</h3><p><a href="https://firebase.google.com/docs/genkit">Firebase Genkit</a> is an open-source framework that helps developers create AI-powered applications. Although developed by the Firebase team, it doesn’t depend on Firebase services, so it can be used on any platform including AWS, GCP, or Azure.</p><p>There are various LLMs available recently, and using them appropriately for different purposes can yield better results. Typically, to execute an LLM, you either directly call a REST API or use an SDK. For example, you might use <a href="https://ai.google.dev/">Gemini</a> for input A and <a href="https://openai.com/api/">OpenAI</a> for input B, depending on the use case.</p><p>Firebase Genkit makes it easy to switch between these use cases. It also provides a developer UI, offering an all-in-one solution for developers to adjust LLM parameters, debug, and troubleshoot.</p><h3>Using Firebase Genkit with Go</h3><p>The final product is available in the following repository, so please refer to it:</p><p><a href="https://github.com/yukinagae/genkit-golang-sample">GitHub - yukinagae/genkit-golang-sample: sample LLM application with Firebase Genkit using Golang</a></p><h3>Setup</h3><ul><li>Go: Install Go by following <a href="https://go.dev/doc/install">Go — Download and Install</a>.</li><li>Genkit: Install Genkit by following <a href="https://firebase.google.com/docs/genkit-go/get-started-go">Firebase Genkit — Getting Started</a>.</li></ul><p>Check the versions of Go and Genkit with the following commands:</p><pre>$ go version<br>go version go1.23.2 darwin/arm64<br>$ genkit --version<br>0.9.1</pre><p>You can either create your own Genkit project based on the official documentation <a href="https://firebase.google.com/docs/genkit-go/get-started-go">Get started with Genkit using Go (alpha)</a>, modify the code, or clone the sample project:</p><pre>$ git clone https://github.com/yukinagae/genkit-golang-sample</pre><h3>Running Locally</h3><p>This example uses <a href="https://ai.google.dev/">Gemini</a>, so obtain an API key in advance.</p><p>You can set the API key as an environment variable and start the Genkit server with the following commands:</p><pre>$ export GOOGLE_GENAI_API_KEY=your_api_key<br>$ genkit start -o -- go run main.go</pre><p>The -o option automatically opens a browser, allowing you to access the developer UI at <a href="http://localhost:4000/">http://localhost:4000</a> to input prompts and debug.</p><p>Issues for the developer UI are managed on <a href="https://github.com/firebase/genkit">firebase/genkit</a>, and development seems active, so it may improve significantly in the future. For more on the developer UI and basic Genkit usage, please refer to my previous article.</p><p><a href="https://medium.com/@yukinagae/your-first-guide-to-getting-started-with-firebase-genkit-6948d88e8a92">Your First Guide to Getting Started with Firebase Genkit</a></p><h3>Code Explanation</h3><ol><li><strong>Initializing Google API Plugin and Getting the Model</strong></li></ol><p>Each LLM is provided as a plugin, and can be used by initializing and obtaining the model as follows. This example obtains the Gemini model:</p><pre>import &quot;github.com/firebase/genkit/go/plugins/googleai&quot;<br><br>// Initialize the Google AI plugin<br>if err := googleai.Init(ctx, nil); err != nil {<br>    log.Fatalf(&quot;Failed to initialize Google AI plugin: %v&quot;, err)<br>}<br><br>// Get the model<br>model := googleai.Model(&quot;gemini-1.5-flash&quot;)<br>// or<br>model := googleai.Model(&quot;gemini-1.5-pro&quot;)</pre><p>If you want to use <a href="https://openai.com/api/">OpenAI</a>, import the dedicated plugin and initialize/obtain the model as follows:</p><pre>import &quot;github.com/yukinagae/genkit-go-plugins/plugins/openai&quot;<br><br>// Initialize the OpenAI plugin<br>if err := openai.Init(ctx, nil); err != nil {<br>    log.Fatalf(&quot;Failed to initialize OpenAI plugin: %v&quot;, err)<br>}<br><br>// Get the model<br>model := openai.Model(&quot;gpt-4o-mini&quot;)<br>// or<br>model := openai.Model(&quot;gpt-4o&quot;)</pre><p><strong>2. Defining the Prompt</strong></p><p>Use the dotprompt.Define() function to define the prompt. Here you can specify what processing you want to do with the prompt, input/output schemas, and parameters like Temperature.</p><pre>summarizePrompt, err := dotprompt.Define(&quot;summarizePrompt&quot;,<br>    &quot;First, fetch this link: {{url}}. Then, summarize the content within 20 words.&quot;,<br>    dotprompt.Config{<br>        Model: model,<br>        Tools: []ai.Tool{webLoader},<br>        GenerationConfig: &amp;ai.GenerationCommonConfig{<br>            Temperature: 1,<br>        },<br>        InputSchema:  jsonschema.Reflect(promptInput{}),<br>        OutputFormat: ai.OutputFormatText,<br>    },<br>)<br>if err != nil {<br>    log.Fatalf(&quot;Failed to initialize prompt: %v&quot;, err)<br>}</pre><p>Genkit also supports Function Calling, so you can define tools using the ai.DefineTool() function.</p><pre>// Define the webLoader tool<br>webLoader := ai.DefineTool(<br>    &quot;webLoader&quot;,<br>    &quot;Loads a webpage and returns the textual content.&quot;,<br>    func(ctx context.Context, input promptInput) (string, error) {<br>        return fetchWebContent(input.URL)<br>    },<br>)</pre><p><strong>3. Defining the Flow</strong></p><p>Use the genkit.DefineFlow() function to define a Flow. In Genkit, a Flow is a single function that combines multiple processes, intended to be called as an API. Here, we define a flow that fetches and summarizes content from a URL. In other use cases, flows can represent more complex processes.</p><pre>// Define a flow that fetches a webpage and summarizes its content<br>genkit.DefineFlow(&quot;summarizeFlow&quot;, func(ctx context.Context, input string) (string, error) {<br>    resp, err := summarizePrompt.Generate(ctx,<br>        &amp;dotprompt.PromptRequest{<br>            Variables: &amp;promptInput{<br>                URL: input,<br>            },<br>        },<br>        nil,<br>    )<br>    if err != nil {<br>        return &quot;&quot;, fmt.Errorf(&quot;failed to generate summary: %w&quot;, err)<br>    }<br>    return resp.Text(), nil<br>})</pre><p><strong>4. Starting the Genkit Server</strong></p><p>You can initialize Genkit and start the server using the genkit.Init() function. By default, the server starts on 127.0.0.1:3400.</p><p>You can change settings like the server address by passing &amp;genkit.Options{} as an argument.</p><pre>// Initialize Genkit<br>if err := genkit.Init(ctx, nil); err != nil {<br>    log.Fatalf(&quot;Failed to initialize Genkit: %v&quot;, err)<br>}</pre><p><strong>5. Calling the API</strong></p><p>The API server is running on 127.0.0.1:3400, so let&#39;s access it with curl. If you get a result like this, it&#39;s successful:</p><pre>$ curl -X POST -H &quot;Content-Type: application/json&quot; -d &#39;{&quot;data&quot;: &quot;https://firebase.google.com/docs/genkit-go/get-started-go&quot;}&#39; http://127.0.0.1:3400/summarizeFlow<br>{&quot;result&quot;: &quot;Firebase Genkit for Go is in alpha and helps you build AI features with Go. \n&quot;}</pre><h3>Conclusion</h3><p>Using Firebase Genkit, we were able to develop an LLM application in Go. If you want to know more, reviewthe <a href="https://firebase.google.com/docs/genkit-go/get-started-go">official documentation</a> or explore the <a href="https://github.com/xavidop/awesome-firebase-genkit">awesome list</a> which has many articles, videos, and samples for reference.</p><p>You can join the Firebase Genkit community on <a href="https://discord.gg/qXt5zzQKpc">Discord</a> and provide feedback on <a href="https://github.com/firebase/genkit/issues">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f96c51c34b10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/firebase-developers/firebase-genkit-for-go-developers-a-guide-to-building-llm-applications-f96c51c34b10">Firebase Genkit for Go Developers: A Guide to Building LLM Applications</a> was originally published in <a href="https://medium.com/firebase-developers">Firebase Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI Evaluation with Promptfoo: A Comprehensive Guide]]></title>
            <link>https://medium.com/@yukinagae/generative-ai-evaluation-with-promptfoo-a-comprehensive-guide-e23ea95c1bb7?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e23ea95c1bb7</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[promptfoo]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Tue, 10 Sep 2024 12:47:42 GMT</pubDate>
            <atom:updated>2024-09-10T12:47:42.486Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post, we’ll explore how to use <a href="https://www.promptfoo.dev/">Promptfoo</a>, a test framework designed to evaluate the output of generative AI models.</p><ul><li><a href="https://www.promptfoo.dev/">Build Secure AI Applications | Promptfoo</a></li><li><a href="https://github.com/promptfoo/promptfoo">GitHub - promptfoo/promptfoo: Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.</a></li></ul><h3>Background</h3><p>In the process of developing generative AI features and services, there are various changes such as:</p><ul><li>Changing the generative AI model (e.g., from GPT-4 to Gemini 1.5 Pro, etc.)</li><li>Modifying and improving prompts</li><li>Registering, updating, and adding data for RAG</li></ul><p>These changes are made as needed to improve the functionality of generative AI, but even small changes can significantly alter the output.</p><p>For example, various regressions can occur:</p><ul><li>Example: Changing to a newly released model resulted in lower output quality compared to the previous model</li><li>Example: Modifying a single line in the prompt caused unnecessary output</li><li>Example: Changing the data for RAG resulted in the inability to answer questions that could previously be answered</li></ul><p>Manually checking for regressions with each change is labor-intensive and requires significant time and resources.</p><p>Upon investigation, various generative AI evaluation frameworks exist (not all have been checked).</p><ul><li><a href="https://github.com/langchain-ai/langchain">LangChain</a></li><li><a href="https://github.com/langfuse/langfuse">Langfuse</a></li><li><a href="https://github.com/openai/evals">OpenAI Evals</a></li><li><a href="https://github.com/confident-ai/deepeval">DeepEval</a></li><li><a href="https://github.com/MagnivOrg/prompt-layer-library">PromptLayer</a></li></ul><h3>Promptfoo</h3><p>After trying out a few tools, I found that <a href="https://www.promptfoo.dev/">Promptfoo</a>, the main subject of this article, seems to be user-friendly, so I will introduce it.</p><p>Here are some features that give a good impression:</p><ul><li>Can be completed locally</li><li>Low learning cost (even non-engineers might be able to use it with some effort)<br>- It’s relatively easy to set up the environment since you only need to install the CLI standalone<br>- Can be described in yaml<br>- Basic assert processing is built-in</li><li>Can test in a matrix like prompt x model x test case<br>- Easy to compare prompts and models</li><li>Surprisingly customizable<br>- Assert processing can be written in javascript/python<br>- The target of the test can be flexibly changed with the provider mechanism<br>- Although it is mainly intended for CLI use, it can also be called from an npm package</li></ul><h3>Try Running It</h3><p>The sample created this time is uploaded to the following GitHub repository, so you can try it by cloning it.</p><p><a href="https://github.com/yukinagae/promptfoo-sample">GitHub - yukinagae/promptfoo-sample</a></p><h3>Installation</h3><p>Refer to <a href="https://www.promptfoo.dev/docs/installation/">Promptfoo — Installation</a>.</p><ul><li>Create a directory</li></ul><pre>$ mkdir promptfoo-sample<br>$ cd promptfoo-sample</pre><ul><li>Install the promptfoo CLI</li></ul><pre># Recommended<br>$ npm install -g promptfoo<br># or<br>$ brew install promptfoo</pre><ul><li>Verify the installation</li></ul><p>It is recommended to install the latest version via npm, but for this example, we will use brew.</p><pre>$ which promptfoo<br>/opt/homebrew/bin/promptfoo<br>$ promptfoo --version<br>0.83.2</pre><ul><li>Initialization (For now, select the most minimal option below)<br>- What would you like to do?: 1<br>- Which model providers would you like to use?: Choose later</li></ul><pre>$ promptfoo init<br><br>? What would you like to do? 1<br>  1) Not sure yet<br>  2) Improve prompt and model performance<br>  3) Improve RAG performance<br>  4) Improve agent/chain of thought performance<br>  5) Run a red team evaluation<br><br>? Which model providers would you like to use? (press enter to skip)<br>❯◉ Choose later<br> ◯ [OpenAI] GPT 4o, GPT 4o-mini, GPT-3.5, ...<br> ◯ [Anthropic] Claude Opus, Sonnet, Haiku, ...<br> ◯ [HuggingFace] Llama, Phi, Gemma, ...<br> ◯ Local Python script<br> ◯ Local Javascript script<br> ◯ Local executable<br><br>✅ Wrote promptfooconfig.yaml. Run `promptfoo eval` to get started!</pre><p>The following two files should be created first:</p><ul><li>README.md</li><li>promptfooconfig.yaml</li></ul><h3>Try Testing</h3><p>As described in the newly created README.md, you can run the sample test in the following three steps:</p><ol><li>Set the OpenAI API key as an environment variable</li></ol><pre>$ export OPENAI_API_KEY=&lt;your-openai-api-key&gt;</pre><ol><li>Run the evaluation with the following command:</li></ol><pre>$ promptfoo eval</pre><ol><li>Check the test results in the Web GUI. The following command will automatically open the browser at <a href="http://localhost:15500/eval/:">http://localhost:15500/eval/:</a></li></ol><pre>$ promptfoo view --yes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1RVkKPlSnJnN0X97.png" /><figcaption>Sample evaluation results of Promptfoo — Graph</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TLh8eMuTkMWW0KwI.png" /><figcaption>Sample evaluation results of Promptfoo — Table</figcaption></figure><p><em>Note: The 12 cases executed are due to matrix testing of [prompts (2 patterns)] x [providers (2 patterns)] x [test cases (3 patterns)].</em></p><h3>Check the Test Contents</h3><p>All tests are described in promptfooconfig.yaml.</p><pre>description: &quot;My eval&quot;<br><br>prompts:<br>  - &quot;Write a tweet about {{topic}}&quot;<br>  - &quot;Write a concise, funny tweet about {{topic}}&quot;<br><br>providers:<br>  - &quot;openai:gpt-4o-mini&quot;<br>  - &quot;openai:gpt-4o&quot;<br><br>tests:<br>  - vars:<br>      topic: bananas<br><br>  - vars:<br>      topic: avocado toast<br>    assert:<br>      - type: icontains<br>        value: avocado<br>      - type: javascript<br>        value: 1 / (output.length + 1)<br><br>  - vars:<br>      topic: new york city<br>    assert:<br>      - type: llm-rubric<br>        value: ensure that the output is funny</pre><p>Let’s go through the contents step by step.</p><ul><li>description</li></ul><p>This is the overall description of the test. It will be displayed in the top left corner when viewed in the Web GUI.</p><pre>description: &quot;My eval&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/0*wVjKVdjfeRxQ3kFN.png" /><figcaption>Sample evaluation results of Promptfoo — Location of description</figcaption></figure><ul><li>prompts</li></ul><p>Specify the prompts to be passed to the generative AI. Here, two patterns of prompts are listed.</p><pre>prompts:<br>  - &quot;Write a tweet about {{topic}}&quot;<br>  - &quot;Write a concise, funny tweet about {{topic}}&quot;</pre><ul><li>providers</li></ul><p>Specify the generative AI models to be used. Here, two patterns of models are listed.</p><p>Refer to <a href="https://www.promptfoo.dev/docs/providers/">Promptfoo — Providers</a> for the providers that can be specified.</p><p>For OpenAI, all details are listed in <a href="https://www.promptfoo.dev/docs/providers/openai/">Promptfoo — Providers &gt; OpenAI</a>.</p><pre>providers:<br>  - &quot;openai:gpt-4o-mini&quot;<br>  - &quot;openai:gpt-4o&quot;</pre><ul><li>tests</li></ul><p>You can specify variables with vars and test items with assert.</p><p>There are many test items that can be specified, so please refer to <a href="https://www.promptfoo.dev/docs/configuration/expected-outputs/">Promptfoo — Assertions &amp; metrics</a>.</p><pre>tests:<br>  - vars: # Execute with the variable `topic` set to `bananas` (output only without assert)<br>      topic: bananas<br><br>  - vars: # Execute with the variable `topic` set to `avocado toast`<br>      topic: avocado toast<br>    assert:<br>      - type: icontains # Check that the test result contains the string `avocado`<br>        value: avocado<br>      - type: javascript # Custom checks can be done with `javascript` or `python`<br>        value: 1 / (output.length + 1)<br><br>  - vars: # Execute with the variable `topic` set to `new york city`<br>      topic: new york city<br>    assert:<br>      - type: llm-rubric # Check the output result with another generative AI model (default: gpt-4o)<br>        value: ensure that the output is funny</pre><h3>Advanced Usage</h3><p>There are various examples of usage in the promptfoo GitHub repository, so please refer to them.</p><p><a href="https://github.com/promptfoo/promptfoo/tree/main/examples">promptfoo/examples at main · promptfoo/promptfoo</a></p><p>Additionally, the official documentation is extensive, so you might find what you want to do by looking at the following:</p><p><a href="https://www.promptfoo.dev/docs/configuration/parameters/">Prompts, tests, and outputs | promptfoo</a></p><h3>Caution</h3><p><strong>Promptfoo is primarily designed to operate locally, but please be aware of the following points.</strong></p><p><strong>If you use the share button in the Web GUI or execute </strong><strong>promptfoo share, the test results will be publicly shared via Cloudflare KV provided by Promptfoo (stored for two weeks).</strong></p><p>It is generally recommended to avoid using this feature.</p><blockquote><em>No, Promptfoo operates locally, and all data remains on your machine. The only exception is when you explicitly use the share command, which stores inputs and outputs in Cloudflare KV for two weeks.</em></blockquote><p><a href="https://www.promptfoo.dev/docs/faq/#does-promptfoo-store-llm-inputs-and-outputs">Frequently asked questions | promptfoo</a></p><h3>Conclusion</h3><p>In conclusion, Promptfoo is a user-friendly framework for evaluating generative AI outputs. Its local operation, low learning curve, and customization options make it valuable for both developers and non-engineers. Matrix testing and various assertions ensure thorough evaluation of changes.</p><p>For more detailed examples and advanced configurations, refer to the <a href="https://github.com/promptfoo/promptfoo">Promptfoo GitHub repository</a> and the <a href="https://www.promptfoo.dev/docs/configuration/parameters/">official documentation</a>.</p><p>Happy testing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e23ea95c1bb7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Slack Bot App with Firebase Genkit in just 100 Lines]]></title>
            <link>https://medium.com/firebase-developers/build-a-slack-bot-app-with-firebase-genkit-in-just-100-lines-71d4e49c9e08?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/71d4e49c9e08</guid>
            <category><![CDATA[cloud-functions]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genkit]]></category>
            <category><![CDATA[firebase]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Sun, 18 Aug 2024 14:21:29 GMT</pubDate>
            <atom:updated>2024-11-19T05:32:50.248Z</atom:updated>
            <content:encoded><![CDATA[<p>For engineers who have started learning generative AI, the next challenge is “How do I actually create a practical application?” This Slack bot app is one practical example. Considering that many engineers use Slack in their daily work and are familiar with it as a communication tool, providing an app in the form of a Slack bot is a natural use case for generative AI. And it’s all within 100 lines of code!</p><p>Details about Firebase Genkit and deploying to Firebase Functions have been explained in previous blog posts, so they will be omitted here.</p><ul><li><a href="https://medium.com/@yukinagae/your-first-guide-to-getting-started-with-firebase-genkit-6948d88e8a92">Your First Guide to Getting Started with Firebase Genkit</a></li><li><a href="https://medium.com/@yukinagae/deploying-your-firebase-genkit-application-with-firebase-functions-99c7d0044964">Deploying Your Firebase Genkit Application with Firebase Functions</a></li></ul><p>In this blog post, I will mainly cover the steps and points to consider when creating a Slack bot app with Firebase Genkit.</p><p>Ultimately, you will be able to create an app that responds when you mention the Slack bot, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*3V9yNNFdiVybJboX.png" /></figure><p>For the code and detailed steps, please refer to the following repository.</p><p><a href="https://github.com/yukinagae/genkit-firebase-functions-slack-bolt-sample">GitHub - yukinagae/genkit-firebase-functions-slack-bolt-sample: Sample project for learning Firebase Genkit and deploying to Firebase Functions and Slack bolt app</a></p><p>Please clone the repository locally in advance. By using this code as is, you should be able to create a Slack bot app without any issues.</p><pre>$ git clone https://github.com/yukinagae/genkit-firebase-functions-slack-bolt-sample.git</pre><h3>Code Explanation</h3><p>The only code you need to look at is functions/src/index.ts. Overall, it&#39;s about 100 lines, but excluding imports and configuration, the main processing is as follows.</p><pre>// Flow definition to answer a question<br>const answerFlow = ai.defineFlow(<br>  {<br>    name: &quot;answerFlow&quot;,<br>    inputSchema: z.string(),<br>    outputSchema: z.string(),<br>  },<br>  async (question: string) =&gt; {<br>    const llmResponse = await ai.generate({<br>      prompt: `You are a helpful AI assistant. You are asked: ${question}`,<br>      model: gpt4o, // Specify the model to use for generation<br>      tools: [],<br>      config: {<br>        temperature: 1, // Set the creativity/variation of the response<br>      },<br>    });<br>    return llmResponse.text;<br>  }<br>);<br><br>// Create a slack receiver<br>function createReceiver() {<br>  const receiver = new ExpressReceiver({<br>    signingSecret: process.env.SLACK_SIGNING_SECRET || &quot;&quot;,<br>    endpoints: &quot;/events&quot;,<br>    processBeforeResponse: true,<br>  });<br><br>  const app = new App({<br>    receiver: receiver,<br>    token: process.env.SLACK_BOT_TOKEN,<br>    processBeforeResponse: true,<br>  });<br><br>  app.event(&quot;app_mention&quot;, async ({ event, context, client, say }) =&gt; {<br>    const { bot_id: botId, text: rawInput, channel } = event;<br>    const { retryNum } = context;<br>    const ts = event.thread_ts || event.ts;<br><br>    if (retryNum) return; // skip if retry<br>    if (botId) return; // skip if bot mentions itself<br><br>    // thinking...<br>    const botMessage = await say({<br>      thread_ts: ts,<br>      text: &quot;typing...&quot;,<br>    });<br>    if (!botMessage.ts) return; // skip if failed to send message<br><br>    const input = rawInput.replace(/&lt;@.*?&gt;/, &quot;&quot;).trim(); // delete mention<br>    const answer = await answerFlow(input); // run the flow to get the answer<br><br>    await client.chat.update({<br>      channel,<br>      ts: botMessage.ts as string,<br>      text: answer,<br>    });<br>  });<br>  return receiver;<br>}<br><br>export const slack = onRequest(<br>  { secrets: [&quot;OPENAI_API_KEY&quot;, &quot;SLACK_BOT_TOKEN&quot;, &quot;SLACK_SIGNING_SECRET&quot;] },<br>  async (req, res) =&gt; {<br>    return createReceiver().app(req, res);<br>  }<br>);</pre><p>Let’s actually run it!</p><h3>Prerequisites</h3><p>Ensure the following are installed:</p><ul><li><strong>Node.js</strong> version 22 or later</li><li><strong>npm</strong></li><li><strong>Genkit</strong></li><li><strong>Firebase CLI</strong></li><li><strong>ngrok</strong></li></ul><p>Also, make sure to create a Firebase project in advance. Since we will be using Cloud Functions for Firebase, don’t forget to switch to the Blaze plan (paid).</p><h3>Creating a Slack Bot App</h3><p>Follow these steps to create an app from the Slack management screen.</p><p>Access <a href="https://api.slack.com/apps">Slack — Your Apps</a> and create an app based on a manifest. While you can create it from scratch, using a manifest is easier.</p><p>Replace only [your_app_name] in the JSON below and copy/paste it. The &quot;request_url&quot;: &quot;http://dummy/events&quot;, part will be replaced later, but for now, it&#39;s set to a dummy value, so you can leave it as is.</p><pre>{<br>  &quot;display_information&quot;: {<br>    &quot;name&quot;: &quot;[your_app_name]&quot;<br>  },<br>  &quot;features&quot;: {<br>    &quot;bot_user&quot;: {<br>      &quot;display_name&quot;: &quot;[your_app_name]&quot;,<br>      &quot;always_online&quot;: true<br>    }<br>  },<br>  &quot;oauth_config&quot;: {<br>    &quot;scopes&quot;: {<br>      &quot;bot&quot;: [<br>        &quot;app_mentions:read&quot;,<br>        &quot;channels:history&quot;,<br>        &quot;chat:write&quot;,<br>        &quot;files:read&quot;<br>      ]<br>    }<br>  },<br>  &quot;settings&quot;: {<br>    &quot;event_subscriptions&quot;: {<br>      &quot;request_url&quot;: &quot;http://dummy/events&quot;,<br>      &quot;bot_events&quot;: [&quot;app_mention&quot;]<br>    },<br>    &quot;org_deploy_enabled&quot;: false,<br>    &quot;socket_mode_enabled&quot;: false,<br>    &quot;token_rotation_enabled&quot;: false<br>  }<br>}</pre><p>After installing the Slack app in your workspace:</p><ul><li>Find the Bot User OAuth Token (located in the left sidebar under Features -&gt; OAuth &amp; Permissions). This will be used later as SLACK_BOT_TOKEN.</li><li>Find the Signing Secret (located in the left sidebar under Settings -&gt; Basic Information). This will be used later as SLACK_SIGNING_SECRET.</li></ul><p>Invite the created Slack app to any channel in the target workspace:</p><pre>/invite @[your_app_name]<br><br># For example, if the app name is [`gpt-test`]<br>/invite @gpt-test</pre><h3>Connecting the Slack Bot App from the Local Emulator</h3><p>One reason to use Cloud Functions for Firebase is the convenience of the Emulator for local development. You can efficiently develop by connecting to the Slack app from the Emulator without deploying.</p><p>First, replace the secret information in functions/.secret.local as needed.</p><pre>$ cp -p ./.secret.local.example ./secret.local<br>$ vim ./secret.local<br>OPENAI_API_KEY=your_api_key<br>SLACK_BOT_TOKEN=your_bot_token<br>SLACK_SIGNING_SECRET=your_signing_secret</pre><p>Now, just start the Emulator.</p><pre>$ npm run emulator<br>✔  functions[us-central1-slack]: http function initialized (http://127.0.0.1:5001/[your_project_name]/us-central1/slack).</pre><p>At this point, it is only running locally and not connected to the Slack app. Use ngrok to forward the local port 5001 to a public URL.</p><pre>$ ngrok http 5001<br>Forwarding https://[your_ngrok_id].ngrok-free.app -&gt; http://localhost:5001</pre><p>[your_ngrok_id] will be a random alphanumeric string automatically assigned by ngrok.</p><p>Set this public URL https://[your_ngrok_id].ngrok-free.app in the Slack management screen to connect the local environment with the Slack app.</p><p>Access Event Subscriptions in the Slack management screen and enter the following in the Request URL field. Wait a moment, and if Request URL Verified is displayed, the connection is successful. Save the settings with Save changes.</p><p><a href="https://[your_ngrok_id].ngrok.io/[your_project_name]/us-central1/slack/events">https://[your_ngrok_id].ngrok.io/[your_project_name]/us-central1/slack/events</a></p><p>Let’s test it on the Slack channel.</p><pre>@[your_app_name] hello</pre><p>If you receive a message reply like the one below, it is successful!</p><pre>Hello! How can I assist you today?</pre><h3>Deploying to Firebase Functions</h3><p>During development, you can experiment using the Emulator and ngrok, but for actual operation, let’s deploy to Cloud Functions for Firebase.</p><p>Set the following three secret information in your Firebase project using the firebase functions:secrets:set command.</p><ul><li>OPENAI_API_KEY</li><li>SLACK_BOT_TOKEN</li><li>SLACK_SIGNING_SECRET</li></ul><p>Here’s how it looks in practice:</p><pre>$ firebase functions:secrets:set OPENAI_API_KEY<br>? Enter a value for OPENAI_API_KEY [input is hidden]<br>$ firebase functions:secrets:set SLACK_BOT_TOKEN<br>? Enter a value for SLACK_BOT_TOKEN [input is hidden]<br>$ firebase functions:secrets:set SLACK_SIGNING_SECRET</pre><p>You can verify the secret information you set using the functions:secrets:access command. It&#39;s a good idea to double-check for typos or copy-paste errors.</p><pre>$ firebase functions:secrets:access OPENAI_API_KEY<br>your_api_key<br>$ firebase functions:secrets:access SLACK_BOT_TOKEN<br>your_bot_token<br>$ firebase functions:secrets:access SLACK_SIGNING_SECRET<br>your_signing_secret</pre><p>Now, let’s deploy!</p><pre>$ npm run deploy</pre><p>Replace the URL of the deployed Firebase Functions with the public URL from ngrok.</p><p>In the Slack management screen, change the Request URL under Event Subscriptions to the following:</p><ul><li><a href="https://slack-[your_function_id]-uc.a.run.app/events">https://slack-[your_function_id]-uc.a.run.app/events</a></li></ul><p>You can find [your_function_id] in the Firebase Console under Build -&gt; Functions.</p><p>Let’s send a message to the bot on the Slack channel again!</p><pre>@[your_app_name] hello</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/0*Jvd968HCN2HG0ajZ.png" /></figure><h3>Conclusion</h3><p>Thanks to the power of Cloud Functions for Firebase and Genkit, we were able to implement a Slack chat bot in less than 100 lines of code — impressive!</p><p>As a next step, you could connect your chat bot to a database or other backend systems using <a href="https://firebase.google.com/docs/genkit/rag">Retrieval Augmented Generation (RAG)</a> or <a href="https://firebase.google.com/docs/genkit/tool-calling">tool calling</a>. Check out the Genkit documentation for more details.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71d4e49c9e08" width="1" height="1" alt=""><hr><p><a href="https://medium.com/firebase-developers/build-a-slack-bot-app-with-firebase-genkit-in-just-100-lines-71d4e49c9e08">Build a Slack Bot App with Firebase Genkit in just 100 Lines</a> was originally published in <a href="https://medium.com/firebase-developers">Firebase Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Your Firebase Genkit Application with Firebase Functions]]></title>
            <link>https://medium.com/@yukinagae/deploying-your-firebase-genkit-application-with-firebase-functions-99c7d0044964?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/99c7d0044964</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[genkit]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[firebase-cloud-functions]]></category>
            <category><![CDATA[firebase]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Sat, 20 Jul 2024 13:10:36 GMT</pubDate>
            <atom:updated>2024-07-20T13:10:36.469Z</atom:updated>
            <content:encoded><![CDATA[<p>This post will guide you through deploying your Firebase Genkit application to Firebase Functions step by step.</p><p>Start by cloning the example repository to get a hands-on experience:</p><pre>$ git clone https://github.com/yukinagae/genkit-firebase-functions-sample.git</pre><p><a href="https://github.com/yukinagae/genkit-firebase-functions-sample">GitHub - yukinagae/genkit-firebase-functions-sample: Sample project for learning Firebase Genkit and deploying to Firebase Functions</a></p><p>For those new to Firebase Genkit, I recommend starting with my introductory blog post: <a href="https://medium.com/@yukinagae/your-first-guide-to-getting-started-with-firebase-genkit-6948d88e8a92">Your First Guide to Getting Started with Firebase Genkit</a>. It lays the groundwork for understanding the core concepts of Firebase Genkit.</p><p><a href="https://medium.com/@yukinagae/your-first-guide-to-getting-started-with-firebase-genkit-6948d88e8a92">Your First Guide to Getting Started with Firebase Genkit</a></p><h3>Preparing for Deployment</h3><p>Before deploying your application, complete the following preparatory steps:</p><ol><li><strong>Create a Firebase project</strong>:</li></ol><p>Navigate to the Firebase Console. Click on Create a project and follow the prompts to create a new Firebase project.</p><ol><li><strong>Switch to the Blaze plan</strong>:</li></ol><p>Firebase Functions require the Blaze (Pay as you go) plan for deployment. In the Firebase Console, select your project, then navigate to the left side bar section to change your plan.</p><ol><li><strong>Configure your Firebase project locally</strong>:</li></ol><p>Update the .firebaserc file in your project’s root directory to include your Firebase project name:</p><pre>{<br>  &quot;projects&quot;: {<br>    &quot;default&quot;: &quot;your_project_name&quot;<br>  }<br>}</pre><h3>Local Emulator</h3><p>To facilitate local development and testing of Firebase Functions, use the Firebase Emulator Suite. Follow these steps to run your functions locally:</p><p>To run Firebase Functions locally using the emulator, set your OpenAI API key as an environment variable and start the emulator:</p><pre>$ export OPENAI_API_KEY=your_api_key<br><br>$ npm run emulator<br># or<br>$ GENKIT_ENV=dev firebase emulators:start --inspect-functions</pre><p>To test the function on the emulator, use the following curl command with a valid token (token1234 in this project):</p><pre>$ curl -X POST -H &quot;Content-Type: application/json&quot; \<br>-H &quot;Authorization: Bearer token1234&quot; \<br>-d &#39;{&quot;data&quot;:{&quot;url&quot;:&quot;https://firebase.blog/posts/2024/04/next-announcements/&quot;,&quot;lang&quot;:&quot;English&quot;}}&#39; \<br>http://127.0.0.1:5001/[your_project_name]/us-central1/summarizeFlow<br>{&quot;result&quot;:&quot;Firebase announced new features at Cloud Next &#39;24, including Firestore vector search, Vertex AI SDKs, and public preview of Gemini integration.&quot;}</pre><p>With an invalid token:</p><pre>$  curl -X POST -H &quot;Content-Type: application/json&quot; \<br>-H &quot;Authorization: Bearer invalid_token&quot; \<br>-d &#39;{&quot;data&quot;:{&quot;url&quot;:&quot;https://firebase.blog/posts/2024/04/next-announcements/&quot;,&quot;lang&quot;:&quot;English&quot;}}&#39; \<br>http://127.0.0.1:5001/[your_project_name]/us-central1/summarizeFlow<br>Unauthorized</pre><p>Token verification is implemented through the authPolicy within the onFlow method:</p><pre>authPolicy: {<br>  async policy() {},<br>  // restrict access using Bearer token<br>  async provider(req, res, next) {<br>    const token = req.headers.authorization?.split(/[Bb]earer /)[1];<br>    // dummy token<br>    if (token &amp;&amp; token === &quot;token1234&quot;) {<br>      next();<br>    } else {<br>      throw new Error(&quot;Unauthorized&quot;);<br>    }<br>  },<br>},</pre><p>While this implementation may not be ready for production, it is sufficiently enough for application testing purposes.</p><p>For scenarios where authorization is not necessary, use noAuth() as follows:</p><pre>import { noAuth } from &quot;@genkit-ai/firebase/functions&quot;;<br><br>authPolicy: noAuth(),</pre><h4>Deploy</h4><p>To authenticate with Firebase and access your projects, use the Firebase CLI login command:</p><pre>$ firebase login</pre><p>To keep your OpenAI API key safe when using Firebase Functions, store it as a secret in Google Cloud Secret Manger:</p><pre>$ firebase functions:secrets:set OPENAI_API_KEY<br>? Enter a value for OPENAI_API_KEY [input is hidden]</pre><p>To confirm your OpenAI API key is correctly stored as a secret, use the following command:</p><pre>$ firebase functions:secrets:access OPENAI_API_KEY<br>your_api_key</pre><p>After securing your API key, you’re ready to deploy your application to Firebase Functions:</p><pre>$ npm run deploy</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BlAGiOCa_q0gYPpbmVCOtw.png" /><figcaption>Firebase GUI — your function</figcaption></figure><p>In the Cloud Functions dashboard, you’ll notice that a secret value OPENAI_API_KEY is specified within the VARIABLES section.</p><p><strong>NOTE:</strong> Secret values within Cloud Functions are applied after deployment, not before deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*If_Ga3bR3n3BIjjqhV8-aA.png" /><figcaption>Cloud Functions GUI — variables</figcaption></figure><p>To test your deployed function, execute the curl command below:</p><pre>$  curl -X POST -H &quot;Content-Type: application/json&quot; \<br>-H &quot;Authorization: Bearer token1234&quot; \<br>-d &#39;{&quot;data&quot;:{&quot;url&quot;:&quot;https://firebase.blog/posts/2024/04/next-announcements/&quot;,&quot;lang&quot;:&quot;English&quot;}}&#39; \<br>https://summarizeflow-[your_function_id]-uc.a.run.app<br>{&quot;result&quot;:&quot;Firebase announced new features at Cloud Next &#39;24, including Firestore vector search, Vertex AI SDKs, and public preview of Gemini integration.&quot;}</pre><p>Replace [your_function_id] with your Firebase project value, found in the Firebase Console under the Functions Dashboard.</p><p>For developers eager to explore the integration of Firebase Genkit with Firebase Functions, I highly recommend diving into my example project:</p><p><a href="https://github.com/yukinagae/genkit-firebase-functions-sample">GitHub - yukinagae/genkit-firebase-functions-sample: Sample project for learning Firebase Genkit and deploying to Firebase Functions</a></p><p>This project serves as a practical guide, offering hands-on experience with the combination of Firebase Genkit and Firebase Functions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99c7d0044964" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your First Guide to Getting Started with Firebase Genkit]]></title>
            <link>https://medium.com/@yukinagae/your-first-guide-to-getting-started-with-firebase-genkit-6948d88e8a92?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6948d88e8a92</guid>
            <category><![CDATA[genkit]]></category>
            <category><![CDATA[firebase]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Sat, 13 Jul 2024 12:55:29 GMT</pubDate>
            <atom:updated>2024-11-18T11:08:03.345Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://firebase.google.com/products/genkit">Firebase Genkit</a> is an open-source framework that helps developers create AI-powered applications.</p><p>In this tutorial, we’ll explore Firebase Genkit, demonstrating how it can enhance your development process.</p><p>For those already familiar with Firebase Genkit, I suggest reading my follow-up blog post, <a href="https://medium.com/@yukinagae/deploying-your-firebase-genkit-application-with-firebase-functions-99c7d0044964">Deploying Your Firebase Genkit Application with Firebase Functions</a>, for insights on crafting a production-ready application.</p><p><a href="https://medium.com/@yukinagae/deploying-your-firebase-genkit-application-with-firebase-functions-99c7d0044964">Deploying Your Firebase Genkit Application with Firebase Functions</a></p><h3>Quick Start for the Impatient</h3><p>If you’re eager to dive right into Firebase Genkit without wading through the details, jumpstart your journey with the <a href="https://github.com/yukinagae/genkit-sample">genkit-sample</a> project.</p><p><a href="https://github.com/yukinagae/genkit-sample">GitHub - yukinagae/genkit-sample: Sample project for learning Firebase Genkit</a></p><p>This sample project is your fastest route to experiencing Firebase Genkit in action. Follow these steps to get up and running:</p><p><strong>1. Clone the Repository</strong>: Start by cloning the genkit-sample repository to your local machine.</p><pre>$ git clone https://github.com/yukinagae/genkit-sample.git</pre><p><strong>2. Install Dependencies</strong>: Navigate into the cloned directory and install the necessary dependencies.</p><pre>$ cd genkit-sample<br>$ npm install</pre><p><strong>3. Set Up Your API Key</strong>: Before you can interact with Firebase Genkit, you’ll need to set your OPENAI_API_KEY environment variable.</p><pre>$ export OPENAI_API_KEY=your_api_key</pre><p><strong>4. Launch the Project</strong>: With your API key in place, kickstart the Genkit server.</p><pre>$ npm run genkit</pre><p>This command fires up your project and opens it in your default web browser <a href="http://localhost:4000/">http://localhost:4000</a>. There you have it!</p><h3>A Beginner’s Guide to Exploring Firebase Genkit</h3><p>For those new to Firebase Genkit and eager to learn more about what it can do, this section is just for you. We’ll walk you through some simple steps to get you started with Firebase Genkit, making it easy and fun to explore its features.</p><h3>1. Setting Up Genkit</h3><p>First things first, we need to configure Genkit settings.</p><pre>import { genkit, z } from &quot;genkit&quot;;<br>import { gpt4o, openAI } from &quot;genkitx-openai&quot;;<br><br>// Log debug output to the console.<br>import { logger } from &quot;genkit/logging&quot;;<br>logger.setLogLevel(&quot;debug&quot;);<br><br>const ai = genkit({<br>  // Use the OpenAI plugin with the provided API key.<br>  // Ensure the OPENAI_API_KEY environment variable is set before running.<br>  plugins: [openAI({ apiKey: process.env.OPENAI_API_KEY })],<br>});</pre><p>In the sample project, we leverage the OpenAI model to dynamically generate content.</p><p><strong>Note:</strong> While the OPENAI_API_KEY environment variable is explicitly set in the code snippet above, the OpenAI model is designed to implicitly recognize the OPENAI_API_KEY environment variable.</p><pre>// Implicit usage<br>plugins: [openAI()],<br><br>// Explicit usage<br>plugins: [openAI({ apiKey: process.env.OPENAI_API_KEY })],</pre><h3>2. Creating Your First Flow</h3><p>A “flow” is just a fancy way of saying the steps your app will take to do something cool. We’ll show you how to set up a simple flow that you can build on as you learn more.</p><pre>import { gpt4o, openAI } from &quot;genkitx-openai&quot;;<br><br>export const summarizeFlow = ai.defineFlow(<br>  {<br>    name: &quot;summarizeFlow&quot;,<br>    inputSchema: z.string(),<br>    outputSchema: z.string(),<br>  },<br>  async (content: string) =&gt; {<br>    const llmResponse = await ai.generate({<br>      prompt: `Summarize the ${content} within 20 words.`,<br>      model: gpt4o, // Specify the model to use for generation<br>      tools: [],<br>      config: {<br>        temperature: 1, // Set the creativity/variation of the response<br>      },<br>    });<br><br>    return llmResponse.text;<br>  }<br>);</pre><p><strong>NOTE:</strong> While adhering to the <a href="https://firebase.google.com/docs/genkit/get-started">Firebase Genkit — Get started</a> guide, it suggests that startFlowsServer(); must be invoked at the end of your code. Interestingly, initiating the Genkit server operates successfully even in the absence of this call.</p><h3>3. Launching the Genkit Server</h3><p>With your first flow ready, it’s time to see it in action!</p><p>To launch Firebase Genkit and explore its graphical user interface, execute one of the following commands in your terminal:</p><pre>$ genkit start -o -- npx tsx src/index.ts<br># or<br>$ npm run genkit</pre><p>After running the command, navigate to <a href="http://localhost:4000/">http://localhost:4000</a> in your web browser to interact with the Firebase Genkit GUI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mj9oj53VemcjczPUoiCvFw.png" /></figure><p><strong>Note:</strong> Should you encounter the error message Error: please pass in the API key or set the OPENAI_API_KEY environment variable, it indicates that the OPENAI_API_KEY has not been properly configured.</p><h3>4. Run Your Flow</h3><p>In the guide’s left sidebar, locate the Flow section and select the summarizeFlow flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hs9Dekqkx9CaEyyTlI1_bg.png" /></figure><p>Proceed to input the following prompt into Provide input (JSON)::</p><pre>&quot;Haiku is a type of poetry unique to Japan that expresses natural aesthetics and human emotions in a limited number of characters. The poems typically comprise 17 on (phonetic units similar to syllables) in lines of five, seven and five (5-7-5) syllables. The basic rule is to include only one word to express the season, called a “kigo,” among the seventeen syllables. The kigo are categorized as spring, summer, fall, winter and new year words, and people usually reference a book called the Saijiki, which lists such kigo like a dictionary, when composing haiku.&quot;</pre><p>(This excerpt is sourced from: <a href="https://www.gov-online.go.jp/eng/publicity/book/hlj/html/202205/202205_12_en.html">Matsuo Basho: The Unparalleled Haiku Poet</a>)</p><p>After entering the prompt, click the Run button. You should observe an output similar to the following:</p><pre>Haiku, a Japanese poetry form, expresses nature and emotions in 17 syllables (5-7-5) and includes a seasonal word (kigo).</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v0dcwM8khlcMEB7nsUx1Og.png" /></figure><p>Congratulations, your first flow runs correctly!</p><h3>5. Adding More to Your Flow</h3><p>Once you’re comfortable with your basic flow, we’ll help you add more to it. This could be new steps, or “tools,” to make your app do even more interesting things.</p><p>The step involves defining a new tool, webLoader, which is responsible for fetching web content. This tool is a stateless function that takes a URL as input and returns the textual content of the webpage.</p><pre>const webLoader = ai.defineTool(<br>  {<br>    name: &quot;webLoader&quot;,<br>    description: &quot;Loads a webpage and returns the textual content.&quot;,<br>    inputSchema: z.object({ url: z.string() }),<br>    outputSchema: z.string(),<br>  },<br>  async ({ url }) =&gt; {<br>    // Fetch the content from the provided URL<br>    const res = await fetch(url);<br>    const html = await res.text();<br>    // Load the HTML content into Cheerio for parsing<br>    const $ = cheerio.load(html);<br><br>    // Remove unnecessary elements<br>    $(&quot;script, style, noscript&quot;).remove();<br><br>    // Prefer &#39;article&#39; content, fallback to &#39;body&#39; if not available<br>    return $(&quot;article&quot;).length ? $(&quot;article&quot;).text() : $(&quot;body&quot;).text();<br>  }<br>);</pre><p>After defining the webLoader tool, the next step is to incorporate it into your existing flow, summarizeFlow. This flow is designed to summarize web content, leveraging the webLoader to fetch the content based on a URL input.</p><pre>import { gpt4o, openAI } from &quot;genkitx-openai&quot;;<br><br>export const summarizeFlow = ai.defineFlow(<br>  {<br>    name: &quot;summarizeFlow&quot;,<br>    inputSchema: z.string(),<br>    outputSchema: z.string(),<br>  },<br>  async (url: string) =&gt; {<br>    const llmResponse = await ai.generate({<br>      prompt: `First, fetch this link: &quot;${url}&quot;. Then, summarize the content within 20 words.`,<br>      model: gpt4o, // Specify the model to use for generation<br>      tools: [webLoader], // Include the webLoader tool defined earlier for fetching webpage content<br>      config: {<br>        temperature: 1, // Set the creativity/variation of the response<br>      },<br>    });<br><br>    return llmResponse.text;<br>  }<br>);</pre><p>In the updated flow, you’ll notice a few key modifications that enhance its functionality.</p><p><strong>Argument Modification</strong></p><p>The function’s argument has been updated to better align with its new purpose:</p><pre>// Before modification<br>async (content: string) =&gt; {<br><br>// After modification<br>async (url: string) =&gt; {</pre><p>This change shifts the focus from processing direct content to working with URLs, allowing for dynamic content fetching.</p><p><strong>Prompt Update</strong></p><p>The prompt has been refined to incorporate the new argument type and streamline the task:</p><pre>// Before modification<br>prompt: `Summarize the ${content} within 20 words.`,<br><br>// After modification<br>prompt: `First, fetch this link: &quot;${url}&quot;. Then, summarize the content within 20 words.`,</pre><p><strong>Tool Integration</strong></p><p>A significant addition to the flow is the integration of a new tool, enhancing its capability to fetch and process web content:</p><pre>// Before modification<br>// (No tools specified)<br><br>// After modification<br>tools: [webLoader], // Include the webLoader tool defined earlier for fetching webpage content</pre><p>By incorporating the webLoader tool, the flow now supports fetching content directly from the specified URL, enabling a more versatile and automated process.</p><h3>6. Running Your Enhanced Flow</h3><p>After enhancing your flow with additional features, it’s crucial to test its functionality. You can either restart the Genkit Server or simply refresh your browser if the server is already running.</p><p>Upon refreshing, you’ll observe the webLoader tool now listed in the left sidebar. Clicking on this tool will navigate you to its input page, showcasing one of Genkit GUI&#39;s standout features: the ability to individually run and test each tool&#39;s functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SD6Fo6Ygk33nAWMJLd4r-w.png" /></figure><p>Testing with an invalid URL:</p><pre>{<br>  &quot;url&quot;: &quot;https://not-found&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XUAMTQsGV8-glkHGpU14Ew.png" /></figure><p>As expected, an error occurs.</p><pre>TRPCClientError: fetch failed<br>    at n.from (http://localhost:4000/main-ZS4I5FPL.js:17:525025)<br>    at http://localhost:4000/main-ZS4I5FPL.js:17:529028<br>    at u.invoke (http://localhost:4000/polyfills-QXL6O4PO.js:16:6385)<br>    at Object.onInvoke (http://localhost:4000/chunk-HQANKQXH.js:22:79656)<br>    at u.invoke (http://localhost:4000/polyfills-QXL6O4PO.js:16:6325)<br>    at Y.run (http://localhost:4000/polyfills-QXL6O4PO.js:16:1715)<br>    at http://localhost:4000/polyfills-QXL6O4PO.js:17:554<br>    at u.invokeTask (http://localhost:4000/polyfills-QXL6O4PO.js:16:7010)<br>    at Object.onInvokeTask (http://localhost:4000/chunk-HQANKQXH.js:22:79472)<br>    at u.invokeTask (http://localhost:4000/polyfills-QXL6O4PO.js:16:6931)</pre><p>Conversely, using a valid URL:</p><pre>{<br>  &quot;url&quot;: &quot;https://en.wikipedia.org/wiki/Haiku&quot;<br>}</pre><p>This time, the tool successfully retrieves content, demonstrating its capability to fetch and process web data:</p><pre>&quot;Jump to content\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\n\t\n\t\n\nMain menu\n\t\n\t\n\n\n\t\t\t\t\n\t\t\n\n\t\n\tMain menu\n\tmove to sidebar\n\thide\n\n\n\t\n\n\t\n\t\tNavigation\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\tMain pageContentsCurrent eventsRandom articleAbout WikipediaContact usDonate\n\t\t\n\t\t\n\t\n\n\n\t\n\t\n\n\t\n\t\tContribute\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\tHelpLearn to editCommunity portalRecent changesUpload</pre><p>To further validate the flow’s effectiveness, let’s input a direct URL into the summarizeFlow:</p><pre>&quot;https://en.wikipedia.org/wiki/Haiku&quot;</pre><p>The result is a concise and informative summary, highlighting the flow’s ability to distill key information:</p><pre>Haiku is a traditional Japanese poetic form consisting of three lines with a 5-7-5 syllable structure, often evoking nature.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQrzKtxpdu9e_yMntcBTPw.png" /></figure><p>Finally, to gain insights into how the flow processes your input, click on the Inspect section in the header. The trace will reveal that the flow employs the webLoader tool to fetch the content, showcasing the step-by-step execution of your flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3CdBEn5KFuqyh9rrwARUw.png" /></figure><p>Now that you’re familiar with the basics, feel free to explore and experiment with the flow and tools further. For a practical starting point, consider checking out <a href="https://github.com/yukinagae/genkit-sample">genkit-sample</a> to run Firebase Genkit.</p><p><a href="https://github.com/yukinagae/genkit-sample">GitHub - yukinagae/genkit-sample: Sample project for learning Firebase Genkit</a></p><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6948d88e8a92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GKEからCloud SQLに接続する2つの方法のレイテンシを調べてみた]]></title>
            <link>https://medium.com/google-cloud-jp/gke%E3%81%8B%E3%82%89cloud-sql%E3%81%AB%E6%8E%A5%E7%B6%9A%E3%81%99%E3%82%8B2%E3%81%A4%E3%81%AE%E6%96%B9%E6%B3%95%E3%81%AE%E3%83%AC%E3%82%A4%E3%83%86%E3%83%B3%E3%82%B7%E3%82%92%E8%AA%BF%E3%81%B9%E3%81%A6%E3%81%BF%E3%81%9F-eb1fbd049d56?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb1fbd049d56</guid>
            <category><![CDATA[cloud-sql]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[日本語]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Mon, 02 Sep 2019 01:13:03 GMT</pubDate>
            <atom:updated>2019-09-02T01:13:03.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e0SJxY3dadTcaCm_tkTUSg.jpeg" /></figure><p><strong><em>※本記事の内容および環境構築用のrepositoryは実用レベルの構成・実装ではなく、あくまでも「GKEからCloud SQLにつながった」+「接続方法によってレイテンシに差があるよね」というだけですのでご注意ください。また検証結果も参考値ですので、もし気になる方は自分で試してみることをオススメします（以下が環境構築用repoです）(｀・ω・´)</em></strong></p><p><a href="https://github.com/yukinagae/latency-comparison-of-cloud-sql-connection">yukinagae/latency-comparison-of-cloud-sql-connection</a></p><h3>TL;DR</h3><ul><li>GKEからCloud SQLに接続する方式は主にPrivate IPとCloud SQL Proxyがある</li><li>パフォーマンスを求めるならやはりPrivate IP接続</li><li>Public IPを使いたい場合にはCloud SQL Proxyを使う</li></ul><h4>結果</h4><p>だいたい10〜15分程度検証しただけですが、当然Private IPの方が低レイテンシですね。Proxyの場合にはPublic IP経由（※regionはus-central1を使用。GKEおよびCloud SQLともに同一region）なのでもちろんPrivate IPよりもネットワーク的に遅くなります。そう考えると20msは割とよい方かもしれません。</p><ul><li>Private IP: 平均7ms</li><li>Cloud SQL Proxy: 平均20ms</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NMPrkx728SDFrjxsQyfF6A.png" /><figcaption>Private IP</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*ORDHawyo2h7uIEvJg1EN5w.png" /><figcaption>Cloud SQL Proxy</figcaption></figure><p>ここからは再現手順などの具体的な内容です。</p><h3>背景</h3><p>GKEを使う際のデータベースは様々選択肢あるかもしれませんが、一番馴染み深いRDBをまず使ってみたくなると思います。GCPの中ではCloud SQLを使用することでMySQLもしくはPostgreSQLを選択することができます（※本記事ではMySQLを選んでいますが、PostgreSQLも同様の方法で接続できると思います）</p><p>公式ドキュメントによると、GKEからCloud SQLに接続する方式は主に以下の2つあります。</p><ul><li>Private IP: 同一ネットワーク内のGKEクラスタから直接接続する</li><li>Cloud SQL Proxy: サイドカーのDockerイメージからプロキシ経由で接続</li></ul><p><a href="https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine">Connect from Google Kubernetes Engine | Cloud SQL for MySQL | Google Cloud</a></p><p>接続方式の使い分けに関してはドキュメントにある程度の方針が書いてあります。</p><blockquote>Private IP is the easiest way to connect from Google Kubernetes Engine if your cluster meets the requirements.</blockquote><blockquote>To connect securely to Cloud SQL from Google Kubernetes Engine using a public IP address, you must use the Cloud SQL Proxy.</blockquote><ul><li>Private IP: もし要件が合うなら最も簡単な方法</li><li>Cloud SQL Proxy: Public IPを使用する場合にはCloud SQL Proxyを使う</li></ul><p>簡単に言うと、Private IPとPublic IPのどちらで接続したいかという使い分けになります。ネットワーク遅延を考えると当然Private IPの方を使うべきですが、外部プロジェクトのデータベースにPublic IPで接続したい場合もあると思います。その場合には同一regionであればPublic IPを使用してもそこまで遅延しないかもしれません。</p><p>考えてみても答えがでないので実際に検証してみましょう。</p><p>Private IPとCloud SQL Proxyで接続した場合にどれくらいレイテンシに差があるか簡易的に検証してみます。</p><h3>検証してみる</h3><p>手動で環境を作るのは面倒なので、terraform + k8sで環境構築できるrepositoryを用意したのでこちらを使って説明します。</p><p><a href="https://github.com/yukinagae/latency-comparison-of-cloud-sql-connection">yukinagae/latency-comparison-of-cloud-sql-connection</a></p><h4><strong>前提条件</strong></h4><ul><li>プロジェクトの用意（既存のプロジェクトを利用することもできますが、試した後に削除しやすいので新規にプロジェクト作成をオススメします）</li><li>プロジェクトのbillingが有効であること</li><li>gcloud がインストール済みであること（※もし未インストールであれば <a href="https://cloud.google.com/sdk/docs/quickstarts">Cloud SDK Quickstarts</a> を参考にインストールしてください）</li><li>tfenv をインストール済みであること（※もし未インストールであれば、<a href="https://github.com/tfutils/tfenv">ここ</a>の手順を見てください。Macであれば brew install tfenv でインストールできます）</li><li>kubectl をインストール済みであること（※もし未インストール済みであれば、<a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">ここ</a>の手順を見てください。Macであれば brew install kubernetes-cli でインストールできます）</li></ul><h4>gcloudのアップデートをします</h4><pre>gcloud components update</pre><h4>作成済みのGCPプロジェクトの認証をします</h4><pre>gcloud auth application-default login<br>gcloud config set project [Your Project ID]</pre><h4>必要なサービスのenableをします</h4><pre>gcloud services enable container.googleapis.com<br>gcloud services enable sqladmin.googleapis.com<br>gcloud services enable servicenetworking.googleapis.com</pre><h3>検証環境を構築する</h3><h4>環境構築用のrepositoryをcloneします</h4><pre>git clone <a href="/service/mailto:git@github.com">git@github.com</a>:yukinagae/latency-comparison-of-cloud-sql-connection.git<br>cd latency-comparison-of-cloud-sql-connection</pre><p>repository内のファイル内容は以下のとおりです。</p><pre>[latency-comparison-of-cloud-sql-connection]<br>├── README.md: 環境の構築手順などを記載<br>├── flask-api<br>│   └── 検証用のPythonファイルなど<br>├── images<br>│   └── *.png: 検証結果のキャプチャ画像など<br>├── k8s<br>│   ├── k8s_private_ip.yaml: Private IP接続用のk8sファイル<br>│   └── k8s_proxy.yaml: Cloud SQL Proxy接続用のk8sファイル<br>└── terraform<br>    └── *.tf: GKEやCloud SQLなどの環境構築用</pre><h4>DockerイメージをビルドしてCotainer Regisgryにプッシュする</h4><pre>cd flask-api<br>docker build -t gcr.io/[Your Project ID]/flask-api:latest .<br><br>gcloud auth configure-docker<br>docker push gcr.io/[Your Project ID]/flask-api:latest</pre><p>APIサーバの実装は、以下のようにただ指定のホストに接続してすぐにcloseしているだけです（※実際の実装とは微妙に異なりますが、やっていることは一緒です）</p><pre>@app.route(&#39;/readiness&#39;)<br>def readiness():<br>    # 接続して<br>    conn = pymysql.connect(user=DB_USER,<br>                           password=DB_PASS,<br>                           host=DB_HOST,<br>                           port=3306, db=&#39;mysql&#39;)</pre><pre>    # クローズするだけ<br>    conn.close()</pre><pre>    return jsonify(status=&quot;OK&quot;, route=&quot;readiness&quot;)</pre><h4>Terraformのセットアップ</h4><pre>cd terraform<br>tfenv install<br>terraform init</pre><p>terraform apply のあとに [Your Project ID] を入力し、最後に yes と入力するとCloud SQLおよびGKE周りのすべての環境がセットアップされます（※10分程度かかったりするので気長に待ちます）</p><pre>$ terraform apply<br>var.project<br>  Your Project ID</pre><pre>Enter a value:</pre><h4>Kubernetesのセットアップ</h4><p>作成したGKEクラスタに接続します(｀・ω・´)</p><pre>gcloud container clusters get-credentials my-gke-cluster</pre><p><strong>Cloud SQLへの接続用のサービスアカウントをSecretとして作成する</strong></p><p>この手順はCloud SQL Proxy用です。Public IP経由で接続する際に対象のCloud SQLインスタンスに触るための権限を与えたサービスアカウントを作成します。</p><ul><li>SA NAME: サービスアカウント名</li><li>SA DISPLAY NAME: サービスアカウント表示名</li></ul><p>SA_NAME も SA_DISPLAY_NAME も任意の文字列でよいです（例えば、proxy-db等）</p><pre>gcloud beta iam service-accounts create [SA NAME] \<br> --display-name &quot;[SA DISPLAY NAME]&quot; \<br> --project [Your Project ID]<br><br>gcloud projects add-iam-policy-binding [Your Project ID] \<br>   --member serviceAccount:[SA NAME]@[Your Project ID].iam.gserviceaccount.com \<br>   --role roles/cloudsql.admin<br><br>gcloud iam service-accounts keys create ./k8s/credentials.json \<br>   --iam-account [SA NAME]@[YOur Project ID].iam.gserviceaccount.com</pre><p>作成したcredentialのJSONファイルをKubernetesのSecretとして登録します。</p><pre>cd k8s<br>kubectl create secret generic cloudsql-instance-credentials --from-file=./credentials.json</pre><h4>Kubernetesのデプロイ</h4><p>k8s ディレクトリ内に以下のファイルがあります。</p><ul><li>k8s_private_ip.yaml: Private IP接続用のk8sファイル</li><li>k8s_proxy.yaml: Cloud SQL Proxyse接続用のk8sファイル</li></ul><p>上記各ファイル内の [Your Project ID] などの変数をご自身の環境に合わせて変更してください。</p><p><strong>Private IP接続するk8sのデプロイする場合</strong></p><pre>kubectl apply -f k8s_private_ip.yaml</pre><p><strong>Cloud SQL Proxy接続するk8sのデプロイする場合</strong></p><pre>kubectl apply -f k8s_proxy.yaml</pre><p>以下のようにpodが動作していれば成功です！</p><pre>$ kubectl get pods</pre><pre>NAME                    READY   STATUS    RESTARTS   AGE<br>test-5dc9b4d885-84vs5   1/1     Running   0          10s</pre><p>最後にコネクション接続時のレイテンシをStackdriver Traceで見てみましょう(´∀｀*)</p><p>ブラウザ上のGCPプロジェクトから[Stackdriver Trace] -&gt; [Trace List] で実際にコネクション接続にどれくらい時間がかかっているかわかります。</p><p>自分でブラウザやcurlでアクセスしなくても k8s_private_ip.yaml と k8s_proxy.yaml の以下の設定の periodSeconds で1秒に1アクセスで死活監視をしているのでそのアクセスでレイテンシの結果を確認できます。</p><pre>readinessProbe:<br>  httpGet:<br>    path: /readiness # `/readiness` というパスで死活監視する<br>    port: 5000<br>  initialDelaySeconds: 3<br>  periodSeconds: 1 # 1秒に1アクセスで死活監視</pre><h4>Stackdriver Traceでレイテンシ確認</h4><p>当然Private IPの方が平均レイテンシは低い。Proxyの場合にはPublic IP経由（※ただし同一region内）なのでもちろんPrivate IPよりもネットワーク的に遅くなります。そう考えると20msは割とよい方かも(｀・ω・´)</p><ul><li>Private IP: 平均7ms</li><li>Cloud SQL Proxy: 平均20ms</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NMPrkx728SDFrjxsQyfF6A.png" /><figcaption>Private IP</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*ORDHawyo2h7uIEvJg1EN5w.png" /><figcaption>Cloud SQL Proxy</figcaption></figure><h3>備考</h3><ul><li>Cloud SQL Proxy接続時には、Cloud SQLへの接続用のサービスアカウントのcredentialの期限が切れた際に再接続するので時間がかかってしまう（らしい）のでconnection poolingなどを検討する必要がある（かも）</li></ul><h3>参考資料</h3><ul><li><a href="https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine">Connecting from Google Kubernetes Engine</a></li><li><a href="https://cloud.google.com/sql/docs/mysql/private-ip">Cloud SQL — Private IP</a></li><li><a href="https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/87">CloudSQL Proxy 175% slower than direct connection</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb1fbd049d56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud-jp/gke%E3%81%8B%E3%82%89cloud-sql%E3%81%AB%E6%8E%A5%E7%B6%9A%E3%81%99%E3%82%8B2%E3%81%A4%E3%81%AE%E6%96%B9%E6%B3%95%E3%81%AE%E3%83%AC%E3%82%A4%E3%83%86%E3%83%B3%E3%82%B7%E3%82%92%E8%AA%BF%E3%81%B9%E3%81%A6%E3%81%BF%E3%81%9F-eb1fbd049d56">GKEからCloud SQLに接続する2つの方法のレイテンシを調べてみた</a> was originally published in <a href="https://medium.com/google-cloud-jp">google-cloud-jp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to make your Kickstarter project successful?]]></title>
            <link>https://medium.com/@yukinagae/how-to-make-your-kickstarter-project-successful-9d91bb64c91a?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d91bb64c91a</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[kickstarter]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[crowdfunding]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Sun, 05 May 2019 04:00:17 GMT</pubDate>
            <atom:updated>2019-05-05T04:06:37.299Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcml915jKYYR32BMsRlX-A.jpeg" /></figure><p>In a previous week, I read <a href="https://www.wired.com/story/kickstarter-turns-10/">this WIRED post</a> about Kickstarter, and this is the beginning of why I write this post.</p><p>In this blog post, I’ll take a look at <a href="https://www.kaggle.com/kemical/kickstarter-projects">this Kickstarter dataset</a> and investigate how to make your project successful.</p><p><a href="https://www.kickstarter.com/">Kickstarter</a> is the world’s largest funding platform for creative projects. It is known to be one of the most popular <em>crowdfunding</em> services in the world. According to its official website, 16 million people have backed a project, $4.2 billion has been pledged, and 161,886 projects have been funded ten years since its launch from 2009.</p><p>The word, <em>crowdfunding</em>,<em> </em>became<em> </em>common in modern society. Before this kind of services or systems, we need to save money or ask money for a bank. But we now have these new platforms as one of the ways to collect money for your creative projects.</p><p>Throughout the analysis, the below three questions will be considered:</p><ol><li>Is the number of projects increasing?</li><li>What factors may be associated with the success/failure of the projects?</li><li>How to make your technology project successful? (Because I’m a tech person)</li></ol><h3>Part 1: Is the number of projects increasing?</h3><p>Since the foundation of Kickstarter in 2019, the number of projects has been increased year by year. The first two years had around 10,000 projects and after five years of the foundation in 2015, the number reached the peak as almost 80,000.</p><p>This indicates the phenomena of <em>crowdfunding</em> became popular and also implies the success of Kickstarter itself.</p><p>However, after the peak in 2015, the number of projects started decreasing. I come up with several possible things can be considered as the cause of the fact such as economic crisis, the emergence of competitors so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*Sy6Yt5cBI_4nJenWGFOFsA.png" /></figure><p>The number of projects seems to be decreasing as the above figure. But it is not really a bad thing if we see the pledge of projects per year as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*PIf_traRiwAx0nqiXHQjEQ.png" /></figure><p>While the number of projects has been decreasing after 2015, the total amount of pledge keeps increasing. Good news is that it can mean some projects can get funded more than before.</p><h3>Part 2: What factors may be associated with the success/failure of the projects?</h3><p>If you are supposed to propose a Kickstarter project, you surely want to make your project successful. The below pie chart shows that only 35% of the projects were succeeded and the rest of the projects were either failed, cancelled or suspended. “35%” is not really a disappointing number compared to <a href="https://en.wikipedia.org/wiki/Batting_average_(baseball)">the averages in MLB</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*UW9XnXGPZnglV39SZxNioQ.png" /></figure><p>The average success rate is only 35%, but the Top 5 categories show better success ratios than the average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*yzOs6UEi19Oy8IfGyeg5dg.png" /></figure><p><strong>Top 5</strong></p><ul><li>Dance</li><li>Theatre</li><li>Comics</li><li>Music</li><li>Art</li></ul><p><strong>Bottom 5</strong></p><ul><li>Technology</li><li>Journalism</li><li>Crafts</li><li>Fashion</li><li>Food</li></ul><p>The categories of the projects seem to really matter here because the success ratio of the Dance and Theatre is almost 60%. The worst category is Technology (20%) and they are three times different.</p><p>One possible reason is that the bottom five categories have higher uncertainty than that of the top five categories. For example, if you want to try new technology in your project, there might be much <em>trial-and-error</em> in the process compared to dancing by yourself.</p><p>Also, it will be difficult for backers to imagine what projects are going to do if it comes to technology, journalism so on.</p><h3>Part 3: How to make your technology project successful?</h3><p>As shown in the previous figures, the Technology category is the worst regarding the success ratio.</p><p>But in this category, what kind of projects were especially failed or succeeded? Here we might get some clues to make our projects successful.</p><p><strong>What subcategories were failed in Technology?</strong></p><p>Especially, the below subcategories were failed:</p><ul><li>Apps</li><li>Software</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*GMWqbg96VZR4yMeen1Fu2Q.png" /></figure><p><strong>What subcategories were successful in Technology?</strong></p><p>In the Technology category, the below subcategories were somehow successful:</p><ul><li>Hardware</li><li>Gadgets</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*9nYurG4rRIhvuQ6OAMoLhg.png" /></figure><p>Considering these facts, successful projects provide something tangible such as hardware and gadgets. On the other hand, most of the applications and software projects were failed.</p><p>It is likely to be the projects provide something tangible can be funded easily. A simple solution is hardware or gadgets for your technology project.</p><h3>Summary</h3><p>Based on the analysis in this post, I’ve had the below three findings:</p><ol><li>The number of projects has been decreasing after 2015, but the total amount of pledge has been increasing.</li><li>A category of the projects is one of the keys factors associated with the success/failure of the projects. Dance, Theatre and Comics are the top three successful categories.</li><li>Hardware and gadgets for your technology project to increase the success ratio.</li></ol><p>If you are interested in the analysis, you can check <a href="https://github.com/yukinagae/kaggle-kickstarter-projects">my GitHub repository</a> and re-produce the result.</p><p>Thanks for reading!<br>If you enjoyed this article, feel free to hit that clap button 👏.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d91bb64c91a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Data FusionでAWSのMySQLからBigQueryにつなげてみた]]></title>
            <link>https://medium.com/google-cloud-jp/cloud-data-fusion%E3%81%A7aws%E3%81%AEmysql%E3%81%8B%E3%82%89bigquery%E3%81%AB%E3%81%A4%E3%81%AA%E3%81%92%E3%81%A6%E3%81%BF%E3%81%9F-4dd02782f278?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dd02782f278</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[cloud-data-fusion]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[日本語]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Tue, 30 Apr 2019 10:20:37 GMT</pubDate>
            <atom:updated>2019-04-30T10:20:37.349Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T-AxdIlXCkEE05BX5YgLCA.jpeg" /></figure><p><strong><em>※本記事の内容は実用レベルの構成・実装ではなく、あくまでも「AWS上のMySQLとBigQueryつながったぞー」というだけですのでご注意ください。</em></strong></p><p>ちなみに、<a href="https://medium.com/google-cloud-jp/cloud-data-fusion%E3%81%A7aws%E3%81%AEpostgresql%E3%81%8B%E3%82%89bigquery%E3%81%AB%E3%83%87%E3%83%BC%E3%82%BF%E7%A7%BB%E8%A1%8C%E3%81%99%E3%82%8B-c2e4035e5e02">このPostgreSQLの記事</a>のMySQL版です。重複している手順もありますが、微妙に手順が異なったりしてハマりがちなので今回は記事にしました（例えば、MySQLの場合にはテンプレートがありますが、PostgreSQLの場合には現在テンプレートがありません）</p><p>また、MySQL to BigQueryのテンプレートは元々うまく動作しなかったのですが、<a href="https://stackoverflow.com/questions/55867841/failed-to-deploy-cloud-data-fusion-transfer-data-from-mysql-to-google-bigquery">こちらのstackoverflowで質問</a>したら即座に修正・回答してくれたので今回の記事を書くことができました。Thanks, Ali!</p><h3>よくある問題</h3><ul><li>WebアプリのデータはMySQLにある</li><li>アプリのデータをBigQueryにバックアップしたい</li></ul><p><a href="https://cloud.google.com/data-fusion/">Cloud Data Fusion</a> を使えばいけるのでは？(´∀｀*)</p><h3>今回やること</h3><ul><li>AWS上にあるMySQLからGCPのBigQueryに、Cloud Data Fusionを使ってデータを持ってくる</li></ul><h3>事前に必要なこと</h3><p><a href="https://medium.com/google-cloud-jp/cloud-data-fusion-%E3%82%92%E6%9C%80%E9%80%9F%E3%81%A7%E8%A7%A6%E3%81%A3%E3%81%A6%E3%81%BF%E3%82%8B-70bd0f313aa5">こちら</a>の手順を参照し、以下を準備してください。</p><ul><li>プロジェクトの用意（既存のプロジェクトを利用することもできますが、試した後に削除しやすいので新規にプロジェクト作成をオススメします）</li><li>プロジェクトのbillingが有効であること</li><li>Cloud Data Fusion APIが有効であること</li><li>AWSの <a href="https://us-west-2.console.aws.amazon.com/rds/home?region=us-west-2#GettingStarted:">Amazon RDS</a> でMySQLのDBがセットアップ済みであること（※外部から接続可能な設定であること）</li><li>Cloud Data Fusionのインスタンスが作成済みであること（※10〜20分程度かかります）</li><li>パーミッションが設定済みであること</li></ul><h3>MySQLのJDBCドライバーをセットアップ</h3><p>Cloud Data FusionのGUI画面から、「HUB」をクリックし、<strong>MySQL JDBC Driver</strong> を選択します。 （5.0.8と5.1.39の二つのバージョンありますが、今回は5.1.39の方をセットアップします）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kheQ_Et0etpqDU8rQlQ32w.png" /></figure><p>書いてあるとおりの手順で進めます。<a href="https://medium.com/google-cloud-jp/cloud-data-fusion%E3%81%A7aws%E3%81%AEpostgresql%E3%81%8B%E3%82%89bigquery%E3%81%AB%E3%83%87%E3%83%BC%E3%82%BF%E7%A7%BB%E8%A1%8C%E3%81%99%E3%82%8B-c2e4035e5e02">PostgreSQL版の記事</a>とほぼ同じなので説明を省略します。</p><ol><li>Download MySQL JDBC Driver: Download</li><li>MySQL Driver Plugin: Deploy</li></ol><h3>MySQL to BigQueryのパイプラインを作成する</h3><p>Cloud Data FusionのGUI画面から、「HUB」をクリックし、<strong>Transfer Data From MySQL to Google BigQuery</strong> を選択します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ztCtZC5U4BArrhjC9yyXMg.png" /></figure><p>いくつかのプラグインおよびドライバーをインストールする必要があるようです。しかし、ほとんどの手順はすでに不要なので、最後の4手順のみを実施し、パイプラインを作成します。</p><p>1. Install Google Cloud Plugins: 不要<br>2. Download MySQL JDBC Driver: 一つ前の手順で設定済みなので不要<br>3. Install MySQL Driver Plugin: 一つ前の手順で設定済みなので不要<br>4. Transfer Data From MySQL to Google BigQuery: これだけクリックする</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kLQEFnzyXvfkV6KjMgF_A.png" /></figure><p>好きなパイプライン名を入力 して「Finish」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yD1g94NA4QjnkuP4F5yiLA.png" /></figure><h3>パイプラインを設定する</h3><p>Database（MySQL）とBigQueryの各ノードの設定をします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yeysW_p8zwzTOnehpisTng.png" /></figure><h4><strong>Databaseノードの設定</strong></h4><p>Connection String</p><pre>jdbc:mysql://[MYSQLの対象ホストのエンドポイント（※外部から接続可能）]:3306/[MySQLのデータベース名]</pre><p>Import Query</p><pre>SELECT * FROM [MySQLの対象テーブル名] WHERE $CONDITIONS;</pre><p>Credentials</p><ul><li>Username: AWSのMySQLで設定した値</li><li>Password: AWSのMySQLで設定した値</li></ul><p>Output Schema</p><ul><li>MySQLの対象テーブルのカラム情報</li></ul><h4><strong>BigQueryノードの設定</strong></h4><p>※途中出てくる <strong>auto-detect </strong>という値を使うと自動で値を検知してセットしてくれるので便利です。</p><p>Reference Name</p><ul><li>任意</li></ul><p>Project ID</p><pre>auto-detect</pre><p>Dataset</p><ul><li>[BigQueryの対象データセット名]</li></ul><p>Table</p><ul><li>[BigQueryの対象テーブル名]</li></ul><p>Temporary Bucket Name</p><ul><li>空白のままで問題ない（※自動的に作成されます）</li></ul><p>Service Account File Path</p><pre>auto-detect</pre><p>Schema</p><ul><li>BigQueryの対象テーブルのカラム情報</li></ul><h3>パイプラインのデプロイ</h3><p>最後に右上のバーから「Deploy」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*g6rYr3D-4cw6W956AxGQ_A.png" /></figure><p>デプロイ中です。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*KGa4gjJEvW2zA2OrlKYVyg.png" /></figure><h3>パイプラインの実行</h3><p>デプロイが完了すると以下のようなGUIに画面遷移します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NafdyYmZDdoO5D3Pvifr-w.png" /></figure><p>中央上部の「Run」をクリックして、パイプラインを実行してみましょう。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*AvOSt6Xp-9QwHfgbetKN0g.png" /></figure><p>少し時間がかかりますが、最後にStatusが <strong>Succeeded</strong> になれば成功です！</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/116/1*u5ilmKWWLxp1AL0C0ph17g.png" /></figure><p>BigQueryの対象テーブルにデータが入っているはずです(´∀｀*)</p><h3>参考資料</h3><ul><li><a href="https://medium.com/google-cloud-jp/cloud-data-fusion-%E3%82%92%E6%9C%80%E9%80%9F%E3%81%A7%E8%A7%A6%E3%81%A3%E3%81%A6%E3%81%BF%E3%82%8B-70bd0f313aa5">Cloud Data Fusion を最速で触ってみる — google-cloud-jp — Medium</a></li><li><a href="https://docs.cask.co/cdap/4.0.0/en/hydrator-manual/plugins/sources/database-batchsource.html">Database — Cask Data Application Platform 4.0.0 Documentation</a></li><li><a href="https://stackoverflow.com/questions/55867841/failed-to-deploy-cloud-data-fusion-transfer-data-from-mysql-to-google-bigquery">stackoverflow: Failed to deploy Cloud Data Fusion “Transfer Data From MySQL to Google BigQuery” template pipeline</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dd02782f278" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud-jp/cloud-data-fusion%E3%81%A7aws%E3%81%AEmysql%E3%81%8B%E3%82%89bigquery%E3%81%AB%E3%81%A4%E3%81%AA%E3%81%92%E3%81%A6%E3%81%BF%E3%81%9F-4dd02782f278">Cloud Data FusionでAWSのMySQLからBigQueryにつなげてみた</a> was originally published in <a href="https://medium.com/google-cloud-jp">google-cloud-jp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Data FusionでAWSのPostgreSQLからBigQueryにデータ移行する]]></title>
            <link>https://medium.com/google-cloud-jp/cloud-data-fusion%E3%81%A7aws%E3%81%AEpostgresql%E3%81%8B%E3%82%89bigquery%E3%81%AB%E3%83%87%E3%83%BC%E3%82%BF%E7%A7%BB%E8%A1%8C%E3%81%99%E3%82%8B-c2e4035e5e02?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2e4035e5e02</guid>
            <category><![CDATA[cloud-data-fusion]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[日本語]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Tue, 23 Apr 2019 07:42:15 GMT</pubDate>
            <atom:updated>2019-04-23T07:55:11.161Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cloud Data FusionでAWSのPostgreSQLからBigQueryにつなげてみた</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpT-JZf1lnZhAixk_fP7bQ.jpeg" /></figure><p><strong><em>※本記事の内容は実用レベルの構成・実装ではなく、あくまでも「AWS上のPostgreSQLとBigQueryつながったぞー」というだけですのでご注意ください。</em></strong></p><h3>よくある問題</h3><ul><li>WebアプリのデータはPostgreSQLにある</li><li>アプリのデータをBigQueryにバックアップしたい</li></ul><p><a href="https://cloud.google.com/data-fusion/">Cloud Data Fusion</a> を使えばいけるのでは？(´∀｀*)</p><h3>今回やること</h3><ul><li>AWS上にあるPostgreSQLからGCPのBigQueryに、Cloud Data Fusionを使ってデータを持ってくる</li></ul><h3>事前に必要なこと</h3><p><a href="https://medium.com/google-cloud-jp/cloud-data-fusion-%E3%82%92%E6%9C%80%E9%80%9F%E3%81%A7%E8%A7%A6%E3%81%A3%E3%81%A6%E3%81%BF%E3%82%8B-70bd0f313aa5">こちら</a>の手順を参照し、以下を準備してください。</p><ul><li>プロジェクトの用意（既存のプロジェクトを利用することもできますが、試した後に削除しやすいので新規にプロジェクト作成をオススメします）</li><li>プロジェクトのbillingが有効であること</li><li>Cloud Data Fusion APIが有効であること</li><li>AWSの <a href="https://us-west-2.console.aws.amazon.com/rds/home?region=us-west-2#GettingStarted:">Amazon RDS</a> でPostgreSQLのDBがセットアップ済みであること（※外部から接続可能な設定であること）</li><li>Cloud Data Fusionのインスタンスが作成済みであること（※10〜20分程度かかります）</li><li>パーミッションが設定済みであること</li></ul><h3>PostgreSQL to BigQueryのパイプラインを作成する</h3><h3>PostgreSQLのJDBCドライバーをセットアップ</h3><p>Cloud Data FusionのGUI画面から、「HUB」をクリックし、<strong>PostgreSQL JDBC Driver</strong> を選択します。 （jre7とjre8の2パターンありますが、今回はjre8の方をセットアップします）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kheQ_Et0etpqDU8rQlQ32w.png" /></figure><p>書いてあるとおりの手順で進めます。</p><ol><li>Download PostgreSQL JDBC Driver: Download</li><li>PostgreSQL Driver Plugin: Deploy</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*7T1YetAQ1D1wosQvZMJGTw.png" /></figure><p>手順1の Download PostgreSQL JDBC Driver: Download を行います。PostgreSQLの公式サイトのリンクから postgresql-9.4.1211.jar というJARファイルをダウンロードします。そして「Finish」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NDrxYYEYqJFDe2VmFEExKQ.png" /></figure><p>手順2で、ダウンロードしたJARファイルをアップロードします。そして「Next」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nzCTgVPgt0GLJEsenUBJyg.png" /></figure><p>ここの設定はデフォルトのままで、「Finish」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NCxvcaoF6WSozrahs2g4uQ.png" /></figure><h3>PostgreSQLのDBとの接続確認</h3><p>まず先にPostgreSQLへの接続確認をすることです。通常の手順と少し異なるかもしれませんが、この方法の方が確実にデータソースとの疎通確認ができるのでオススメです。</p><p>Cloud Data FusionのGUI画面の左サイドバーから、「Wrangler」を選択します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*Rfsvsmb3yltZfuawfaG2FA.png" /></figure><p>以下のようにデータソースの一覧画面が表示されます。最下段の「Add Connection」をクリックすると、新たにデータソースを追加することができます。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*ARJvl5BxUnNQPSn2WR5YRw.png" /></figure><p>今回はPostgreSQLと接続するため、「Database」を選択します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*tm_IxWPjVwb2NVkfM6k0cQ.png" /></figure><p><strong>Driver Installed</strong> と表示されている、さきほどセットアップしたPostgreSQLを選択します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*dKnlPhXZ-eaSYi82KI0SfQ.png" /></figure><p>接続設定の画面が表示されるので、AWSのPostgreSQLの設定情報等を入力します（※このPostgreSQLのDBが外部から接続可能な設定にしておいてください。In-boundネットワーク周りの設定が必要だと思います）。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*zC8TktWsdQ7_MTbK1Zxhqw.png" /></figure><p>「Test Connection」をクリックした後に「Database」を選択できるようになります。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*93LQMpzi-s5jTHD5E-eTSw.png" /></figure><p>最後に「Add Connection」をクリックします。</p><p>これで疎通確認は完了です。</p><h3>パイプラインの作成</h3><p>一つ前の手順でテーブルの情報が表示でき、その右側に「Create a Pipeline」というのがあるのでこれをクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*LbV6lNKUn0yUGMLN-BCZVw.png" /></figure><p>「Batch pipeline」を選択します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*8hSl5HboQPckqGpXNwtGvA.png" /></figure><p>これでテンプレートのパイプラインが作成できました！ 右側の「Wrangler」はデータ前処理や変換用のノードなので、今回は不要なので削除してしまいます。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*iWCWqKgbTCvAlaMI1eowrA.png" /></figure><p>「Wrangler」ノードの右下から「Delete」をクリックして削除します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*s4bF17THeyfzH5ljAOfu7A.png" /></figure><p>その代わりに左のサイドバーの「Sink」から<strong>BigQuery</strong>をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*aB8bgzVDJmOsq8nKZikRxg.png" /></figure><p>以下のようにノードを矢印でつなげます。 （※まだBigQueryノード側を設定していないので、黄色でwarningが表示されていますね）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*GtDSBRu63gYjU-mjcqbP9g.png" /></figure><h3>Databaseノードの設定</h3><p>「Database — user」にホバーすると「Properties」が表示されるのでこれをクリックします。</p><p>プロパティ画面が表示されるので、右側の「Output Schema」を入力します（※Import/Exportすることもできます）。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*cmyUqsiwG-oAa7fQkMu47g.png" /></figure><p>例）以下のようにPostgreSQLの入力テーブルのカラムに従って入力します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*wM1MwgyadFzQWGm77N7AZA.png" /></figure><h3>BigQueryノードの設定</h3><p>次にBigQueryノードの設定をします。</p><p>「Basic」の設定部分のみ任意で入力します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SwGb9N6wgJLTdTlMUZNIIg.png" /></figure><h3>パイプラインに名前をつける</h3><p>左上辺りのバーからパイプラインの名前を設定します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aplF56lPwB_qiOFMVa0n_w.png" /></figure><h3>パイプラインのデプロイ</h3><p>最後に右上のバーから「Deploy」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*g6rYr3D-4cw6W956AxGQ_A.png" /></figure><p>デプロイ中です。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*KGa4gjJEvW2zA2OrlKYVyg.png" /></figure><h3>パイプラインの実行</h3><p>デプロイが完了すると以下のGUIに画面遷移します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*azqTKaiUnvVSxMWAHXbqgQ.png" /></figure><p>中央上部の「Run」をクリックして、パイプラインを実行してみましょう。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*AvOSt6Xp-9QwHfgbetKN0g.png" /></figure><p>少し時間がかかりますが、最後にStatusが <strong>Succeeded</strong> になれば成功です！</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/116/1*u5ilmKWWLxp1AL0C0ph17g.png" /></figure><p>BigQueryの対象テーブルにデータが入っているはずです(´∀｀*)</p><h3>参考資料</h3><ul><li><a href="https://medium.com/google-cloud-jp/cloud-data-fusion-%E3%82%92%E6%9C%80%E9%80%9F%E3%81%A7%E8%A7%A6%E3%81%A3%E3%81%A6%E3%81%BF%E3%82%8B-70bd0f313aa5">Cloud Data Fusion を最速で触ってみる — google-cloud-jp — Medium</a></li><li><a href="https://docs.cask.co/cdap/4.0.0/en/hydrator-manual/plugins/sources/database-batchsource.html">Database — Cask Data Application Platform 4.0.0 Documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2e4035e5e02" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud-jp/cloud-data-fusion%E3%81%A7aws%E3%81%AEpostgresql%E3%81%8B%E3%82%89bigquery%E3%81%AB%E3%83%87%E3%83%BC%E3%82%BF%E7%A7%BB%E8%A1%8C%E3%81%99%E3%82%8B-c2e4035e5e02">Cloud Data FusionでAWSのPostgreSQLからBigQueryにデータ移行する</a> was originally published in <a href="https://medium.com/google-cloud-jp">google-cloud-jp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Data Fusion を最速で触ってみる]]></title>
            <link>https://medium.com/google-cloud-jp/cloud-data-fusion-%E3%82%92%E6%9C%80%E9%80%9F%E3%81%A7%E8%A7%A6%E3%81%A3%E3%81%A6%E3%81%BF%E3%82%8B-70bd0f313aa5?source=rss-63b3f8a4e2d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/70bd0f313aa5</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-data-fusion]]></category>
            <category><![CDATA[日本語]]></category>
            <dc:creator><![CDATA[Yuki Nagae]]></dc:creator>
            <pubDate>Thu, 11 Apr 2019 07:04:22 GMT</pubDate>
            <atom:updated>2019-04-11T13:59:55.443Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hR6JPE9uxBYhfA3vVaUrkg.jpeg" /></figure><h3>Cloud Data Fusion とは？</h3><p><a href="https://cloud.google.com/data-fusion/">Cloud Data Fusion</a> とはGoogle Cloud Next 2019でアナウンスされた、フルマネージドのデータ統合プラットフォームです。</p><p>今回の新サービスに関する情報は以下の公式ブログが詳しいです。</p><p>参照: <a href="https://cloud.google.com/blog/products/data-analytics/google-cloud-smart-analytics-accelerates-your-business-transformation">From data ingestion to insight prediction: Google Cloud smart analytics accelerates your business transformation | Google Cloud Blog</a></p><h3>今回やること</h3><p><a href="https://cloud.google.com/data-fusion/docs/quickstart">Quickstart | Cloud Data Fusion</a>の通りにやってみます。</p><p>内容としては</p><ol><li>Cloud StorageにあるJSONファイル（本のベストセラーに関するデータ）を読み込む</li><li>データをパースして綺麗にする</li><li>先週出版されて評価が高かった$25以下の本のデータをBigQueryに投入</li></ol><p>です。</p><h3>事前に必要なこと</h3><ul><li>プロジェクトの用意（既存のプロジェクトを利用することもできますが、試した後に削除しやすいので新規にプロジェクト作成をオススメします）</li><li>プロジェクトのbillingが有効であること</li><li>Cloud Data Fusion APIが有効であること</li></ul><h3>インスタンスの作成</h3><p>Cloud Data Fusionのページにアクセスし、「CREATE AN INSTANCE」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ikQjnecfkpNv5toVDjvMeA.png" /></figure><p>適当な Instance name を入力して「CREATE」します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Z3i8kNcX8ZsRIFyUNC4yQ.png" /></figure><p>インスタンス作成に20分程度かかるらしいのでゆっくり待ちましょう。 （実際、13分かかりました）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-xbPTeq5ett-KIunaXf3A.png" /></figure><h3>パーミッションを設定する</h3><p>「Instance name（上記例では data-fusion-1 ）」をクリックし、「Service Account」の値を確認。</p><p>以下の場合だと、cloud-datafusion-management-sa@yd9c10377068300f0-tp.iam.gserviceaccount.com です。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*3YYWiuOlSZWAHv_xQIoM7Q.png" /></figure><p>IAM &amp; adminサービスでパーミッションを設定します。以下画面で「ADD」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*ZglTv5pWjhz5ppU76InqPA.png" /></figure><ul><li>「New members」に先程確認した「Service Account」の値を設定します。</li><li>この場合は、 cloud-datafusion-management-sa@yd9c10377068300f0-tp.iam.gserviceaccount.com です。</li><li>「Role」には Cloud Data Fusion API Service Agent を設定します。</li><li>「Save」をクリックします。</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*Pl-ZF_Qhc9puVkqhvNxLkw.png" /></figure><p>注: これの手順を忘れていたせいで30分ほど時間を無駄にしてしまった。どのようなエラーになるかは最下段の「おまけのエラー1」を参照してください。ドキュメントをちゃんと読むことが大事ですね。</p><h3>IPのQuotaの上限を増やす</h3><p>この手順は公式ドキュメントに記載ないです。この手順を行うとご使用のメールアドレスに確認メールがきますが、その内容によると申請が通るのに早くても2〜3日かかるらしいですが、運がよければ数時間以内に結果がきます。（※もしこの手順がめんどくさいユーザーは、後述の「（おまけ手順）「IPのQuotaの上限を増やす」の申請がめんどくさいユーザー向け」を実施してください）</p><ol><li>IAM &amp; adminサービスから「Quota」を選択</li><li>対象のService（※この場合は Compute Engine API（In-use IP addresses） で、かつインスタンス作成したlocationである us-west1を選択）</li><li>「EDIT QUOTAS」をクリックします</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*f1a93_NBqyFhg9x-tgNwvQ.png" /></figure><p>ほしいquotaの数を入力し、「Submit request」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*uHQRk9moWS14UEZGSDiyHw.png" /></figure><p>注: これの手順がないとどのようなエラーになるかは最下段の「おまけのエラー2」を参照してください。</p><h3>Cloud Data FusionのGUIを見てみる</h3><p>Cloud Data Fusionサービスに戻り、「Instance name（上記例では data-fusion-1 ）」をクリックし、Instance URLの「View Instance」をクリックするとGUIに遷移します。</p><p>初回なので何もありませんね。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VeijiieJOgfCcTdsc--P0g.png" /></figure><h3>サンプルのpipelineをデプロイする</h3><p>すでに Cloud Data Fusion Hub という仕組みでpipelineが共有されているみたいなので、そのサンプルをそのまま使います。</p><p>Cloud Data FusionのGUI画面から、「HUB」をクリックします。</p><p>色々選べるみたいですね。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wUR8IYz-YW4XOy9iaGHX2A.png" /></figure><p>左のサイドバーの「Pipelines」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2hWKxN9ZdSZQqLJvuU9hrA.png" /></figure><p>Cloud Data Fusion Quickstart を選び、「Create」をクリックします。</p><p><em>BEST SELLER</em> と書いてあるとおり、本のベストセラーデータのようです。</p><p>pipelineに適当な名前をつけ、「Finish」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLX1nQyZ3pcIXxWNtagpAQ.png" /></figure><p>pipelineができました！ 次に「Customize Pipeline」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNVqFcPN9hXMYPsv9XG_EA.png" /></figure><p>GUIのクールな画面が出てきましたね。この画面のことを <strong>Data Fusion Studio</strong> と呼びます。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AlcvrezkJUcOx_jgya634g.png" /></figure><p>pipelineは作成済みですが、まだデプロイされていません。右上の「Deploy」をクリックしましょう。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bOSfUYUSlexdibuetm1oaA.png" /></figure><p>デプロイ完了しました。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNWsRldlzhCtRAfzG5nDlQ.png" /></figure><h3>（おまけ手順）「IPのQuotaの上限を増やす」の申請がめんどくさいユーザー向け</h3><p>「IPのQuotaの上限を増やす」の申請がめんどくさいユーザーは以下の設定のworker数を減らせば大丈夫です。</p><ol><li>Data Fusion Studio上部の「Configure」をクリックします</li><li>「Compute config」から「Customize」をクリックします</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*4bNZEexZM162fq5kGx8WdA.png" /></figure><p>Worker Nodesの「Number of Workers」の値を小さい数字（※7以下の値なら問題ないはずです。Master NodeとWorker Nodesの総計が、許可されているIP数以下になるようにします。本記事では許可されているIPは8を想定しています）を設定し、「Done」をクリックします。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*0kwYfQH0LjfWaJdNuU9YKg.png" /></figure><h3>pipelineを実行する</h3><p>準備は万端なので、試しに実行してみましょう。上部の「Run」をクリックします。 実行すると、 Cloud Dataprocクラスターがプロビジョニングされてpipelineが実行されるようです。 （Apache Hadoop MapReduce もしくは Apache Spark選んで使用することができます）</p><p>「Status」が Succeeded になっていれば成功です！</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Owbdh1dIpdkT9Tnt0jAgUg.png" /></figure><h3>BigQueryで結果を確認する</h3><p>結果をBigQueryで確認しましょう。以下のようなテーブルが新規作成されているはずです。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*Ves1nQqjSyGc7BjbdeP20g.png" /></figure><p>プレビューを見ると、以下のような結果が格納されています。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*Ohpbs2vX10831DuRQzBJ4g.png" /></figure><p>Good job!</p><h3>おまけのエラー1</h3><p>「パーミッションを設定する」の手順を忘れていたせいで、pipelineの「Run」でエラーになってしまった。 その一部始終をご覧ください。</p><p>あれ、Failedしてしまいました汗。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yfnIaZ7jag7x6eYeYhXEog.png" /></figure><p>Logsを見てみましょう。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zoSxt9cVjFc32QOeaX66ug.png" /></figure><p>以下のエラーを見ると、「compute.firewalls.list」のpermissionがないためにエラーになっているように見えます。</p><pre>com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden<br>{<br>  &quot;code&quot; : 403,<br>  &quot;errors&quot; : [ {<br>    &quot;domain&quot; : &quot;global&quot;,<br>    &quot;message&quot; : &quot;Required &#39;compute.firewalls.list&#39; permission for &#39;projects/cloud-data-fusion-test&#39;&quot;,<br>    &quot;reason&quot; : &quot;forbidden&quot;<br>  } ],<br>  &quot;message&quot; : &quot;Required &#39;compute.firewalls.list&#39; permission for &#39;projects/cloud-data-fusion-test&#39;&quot;<br>}</pre><p>原因がわからず30分ほど右往左往する。エラーメッセージでgoogle検索しても原因がわからない。</p><h3>おまけのエラー2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zpw9HcLzXYRC4En9livMuQ.png" /></figure><p>別のエラー: PROVISION task failed in REQUESTING_CREATE state for program run program_run:default.DataFusionQuickstart.-SNAPSHOT.workflow.DataPipelineWorkflow.05950624–5c12–11e9–8aba-129dddfa0155.</p><pre>com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Insufficient &#39;IN_USE_ADDRESSES&#39; quota. Requested 11.0, available 8.0.<br>  at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49) ~[na:na]<br>  at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) ~[na:na]<br>  at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) ~[na:na]<br>  at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:95) ~[na:na]<br>  at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) ~[na:na]<br>  at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) ~[com.google.guava.guava-13.0.1.jar:na]<br>  at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) ~[na:na]<br>  at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) ~[com.google.guava.guava-13.0.1.jar:na]<br>  at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) ~[com.google.guava.guava-13.0.1.jar:na]<br>  at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) ~[com.google.guava.guava-13.0.1.jar:na]<br>  at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:492) ~[na:na]<br>  at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:467) ~[na:na]<br>  at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) ~[na:na]<br>  at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) ~[na:na]<br>  at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) ~[na:na]<br>  at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:392) ~[na:na]<br>  at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:475) ~[na:na]<br>  at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) ~[na:na]<br>  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:557) ~[na:na]<br>  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:478) ~[na:na]<br>  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:590) ~[na:na]<br>  at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[na:na]<br>  at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[na:na]<br>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]<br>  at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]<br>  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_212]<br>  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_212]<br>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]<br>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]<br>  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]<br>Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Insufficient &#39;IN_USE_ADDRESSES&#39; quota. Requested 11.0, available 8.0.<br>  at io.grpc.Status.asRuntimeException(Status.java:526) ~[na:na]<br>  ... 19 common frames omitted</pre><p>IAM &amp; adminの「Quota」を確認（※us-west1 のlocationだけ確認）すると、Limitが8になっています。</p><p>つまり、Requested 11.0, available 8.0. というエラーの意味は、IPを11個要求したが使用可能なIPが8個しかないということです。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*7ICVPeKUVaygXNUY55tpHA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70bd0f313aa5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud-jp/cloud-data-fusion-%E3%82%92%E6%9C%80%E9%80%9F%E3%81%A7%E8%A7%A6%E3%81%A3%E3%81%A6%E3%81%BF%E3%82%8B-70bd0f313aa5">Cloud Data Fusion を最速で触ってみる</a> was originally published in <a href="https://medium.com/google-cloud-jp">google-cloud-jp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>