<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Berhan Ay on Medium]]></title>
        <description><![CDATA[Stories by Berhan Ay on Medium]]></description>
        <link>https://medium.com/@berhanay?source=rss-feb0d1d6abf2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DHmrrtzMmgPbiOkY8M04iw@2x.jpeg</url>
            <title>Stories by Berhan Ay on Medium</title>
            <link>https://medium.com/@berhanay?source=rss-feb0d1d6abf2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 11:52:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@berhanay/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[☁️ Getting Started with Huawei Cloud SDK Core for Python: A Comprehensive Guide]]></title>
            <link>https://medium.com/huawei-developers/%EF%B8%8F-getting-started-with-huawei-cloud-sdk-core-for-python-a-comprehensive-guide-727acc7668ad?source=rss-feb0d1d6abf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/727acc7668ad</guid>
            <category><![CDATA[huawei-cloud]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sdk]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[huawei]]></category>
            <dc:creator><![CDATA[Berhan Ay]]></dc:creator>
            <pubDate>Fri, 14 Mar 2025 13:35:46 GMT</pubDate>
            <atom:updated>2025-03-14T13:35:46.121Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Yp-6r5-8m3nOP3F.jpg" /><figcaption>Banner Photo</figcaption></figure><h3>📚 Introduction</h3><p>In today’s cloud-driven world, automation and programmatic management of cloud resources have become essential skills for developers and DevOps engineers. Huawei Cloud, a rapidly growing cloud service provider, offers a comprehensive SDK that allows developers to interact with their services using Python. Whether you’re managing cloud servers, configuring networking, or utilizing AI services, the Huawei Cloud SDK for Python provides a streamlined way to interact with Huawei Cloud’s extensive service portfolio.</p><p>The Huawei Cloud Python SDK enables you to easily work with various Huawei Cloud services such as Elastic Compute Service (ECS) and Virtual Private Cloud (VPC) without the complexity of handling API-related tasks directly. It abstracts away the low-level details of making HTTP requests, authentication, error handling, and response parsing, allowing you to focus on building your applications.</p><p>This guide will walk you through everything you need to know about the Huawei Cloud SDK Core for Python — from installation and authentication to making your first API calls and implementing advanced features. By the end of this article, you’ll be equipped with the knowledge to automate your Huawei Cloud infrastructure effectively.</p><h3>Table of Contents</h3><ol><li><a href="#prerequisites">Prerequisites</a></li><li><a href="#installation">Installation</a></li><li><a href="#authentication">Authentication</a></li><li><a href="#making-your-first-api-call">Making Your First API Call</a></li><li><a href="#working-with-http-configuration">Working with HTTP Configuration</a></li><li><a href="#error-handling">Error Handling</a></li><li><a href="#asynchronous-operations">Asynchronous Operations</a></li><li><a href="#logging">Logging</a></li><li><a href="#advanced-features">Advanced Features</a></li><li><a href="#best-practices">Best Practices</a></li><li><a href="#summary">Summary</a></li></ol><h3>Prerequisites</h3><p>Before diving into the Huawei Cloud SDK, ensure you have:</p><ul><li>Python 3.3 or later installed (run python --version to check)</li><li>A Huawei Cloud account with real-name authentication</li><li>Access Key (AK) and Secret Key (SK) of your Huawei Cloud account</li><li>Project ID for the region where you want to operate</li><li>Activated the specific services you plan to use in the Huawei Cloud console</li></ul><p>You can find your AK and SK in the Huawei Cloud console under “My Credentials” &gt; “Access Keys”. For project ID, navigate to “My Credentials” &gt; “API Credentials” in the console.</p><h3>Installation</h3><p>There are several ways to install the Huawei Cloud SDK for Python. You can install either individual service SDKs or the complete collection package.</p><h3>Using pip (Recommended)</h3><p>Installing the Huawei Cloud SDK Core for Python is straightforward using pip:</p><pre># Install the core library<br>pip install huaweicloudsdkcore</pre><h4>Individual Service Libraries</h4><p>Depending on which Huawei Cloud services you want to use, you’ll need to install specific service libraries. For example, to use the VPC service:</p><pre># Install the VPC service library<br>pip install huaweicloudsdkvpc</pre><p>For other services, replace “vpc” with the appropriate service name (e.g., “ecs” for Elastic Cloud Server, “live” for Live streaming service, etc.).</p><h4>Complete SDK Collection</h4><p>Alternatively, you can install all supported service packages at once:</p><pre># Install all service libraries<br>pip install huaweicloudsdkall</pre><h3>Using Source Code</h3><p>If you prefer to install from source code:</p><pre># Install the core library<br>cd huaweicloudsdkcore-${version}<br>python setup.py install<br><br># Install specific service libraries<br>cd huaweicloudsdkservice-${version}<br>python setup.py install</pre><p>You can get the SDK version information through <a href="https://console-intl.huaweicloud.com/apiexplorer/#/sdkcenter?language=Python">SDK center</a> or <a href="https://pypi.org/search/?q=huaweicloudsdk">PyPI</a>.</p><h3>Authentication</h3><p>There are multiple ways to authenticate with the Huawei Cloud SDK. Let’s explore the authentication types and management methods.</p><h3>Types of Credentials</h3><p>There are two types of Huawei Cloud services: regional services and global services. Global services include BSS, DevStar, EPS, IAM, RMS, and TMS. The authentication method differs slightly between them:</p><ul><li>For regional services: projectId is required (BasicCredentials)</li><li>For global services: domainId is required (GlobalCredentials)</li></ul><h3>Authentication Methods</h3><h4>1. Permanent AK/SK</h4><p>This is the most common method:</p><pre>from huaweicloudsdkcore.auth.credentials import BasicCredentials, GlobalCredentials<br># For regional services<br>credentials = BasicCredentials(ak, sk, project_id)<br># For global services<br>credentials = GlobalCredentials(ak, sk, domain_id)</pre><h4>2. Temporary AK/SK</h4><p>For temporary access using a security token:</p><pre>from huaweicloudsdkcore.auth.credentials import BasicCredentials, GlobalCredentials<br># For regional services<br>credentials = BasicCredentials(ak, sk, project_id).with_security_token(security_token)<br># For global services<br>credentials = GlobalCredentials(ak, sk, domain_id).with_security_token(security_token)</pre><h4>3. IdpId &amp; IdTokenFile</h4><p>For federated authentication using OpenID Connect:</p><pre>from huaweicloudsdkcore.auth.credentials import BasicCredentials, GlobalCredentials<br><br># For regional services<br>credentials = BasicCredentials().with_idp_id(idp_id).with_id_token_file(id_token_file).with_project_id(project_id)<br># For global services<br>credentials = GlobalCredentials().with_idp_id(idp_id).with_id_token_file(id_token_file).with_domain_id(domain_id)</pre><h3>Authentication Management</h3><p>Starting from SDK version v3.0.98, you can obtain credentials from various providers:</p><h4>Environment Variables</h4><p>For AK/SK authentication:</p><pre># Linux<br>export HUAWEICLOUD_SDK_AK=YOUR_AK<br>export HUAWEICLOUD_SDK_SK=YOUR_SK<br>export HUAWEICLOUD_SDK_PROJECT_ID=YOUR_PROJECT_ID  # for regional services<br>export HUAWEICLOUD_SDK_DOMAIN_ID=YOUR_DOMAIN_ID    # for global services</pre><p>For IdpId/IdTokenFile authentication:</p><pre># Linux<br>export HUAWEICLOUD_SDK_IDP_ID=YOUR_IDP_ID<br>export HUAWEICLOUD_SDK_ID_TOKEN_FILE=/path/to/token_file<br>export HUAWEICLOUD_SDK_PROJECT_ID=YOUR_PROJECT_ID  # for regional services<br>export HUAWEICLOUD_SDK_DOMAIN_ID=YOUR_DOMAIN_ID    # for global services</pre><p>Then in your code:</p><pre>from huaweicloudsdkcore.auth.provider import EnvCredentialProvider<br><br># For regional services<br>provider = EnvCredentialProvider.get_basic_credential_env_provider()<br>credentials = provider.get_credentials()<br># For global services<br>provider = EnvCredentialProvider.get_global_credential_env_provider()<br>credentials = provider.get_credentials()</pre><h4>Profile</h4><p>You can store credentials in profile files:</p><pre>[basic]<br>ak = your_ak<br>sk = your_sk<br>project_id = your_project_id<br><br>[global]<br>ak = your_ak<br>sk = your_sk<br>domain_id = your_domain_id</pre><pre>from huaweicloudsdkcore.auth.provider import ProfileCredentialProvider<br># For regional services<br>provider = ProfileCredentialProvider.get_basic_credential_profile_provider()<br>credentials = provider.get_credentials()</pre><h4>Metadata (for ECS instances)</h4><pre>from huaweicloudsdkcore.auth.provider import MetadataCredentialProvider<br>provider = MetadataCredentialProvider.get_basic_credential_metadata_provider()<br>credentials = provider.get_credentials()</pre><h4>Provider Chain</h4><p>The SDK can try multiple providers in sequence:</p><pre>from huaweicloudsdkcore.auth.provider import CredentialProviderChain<br>chain = CredentialProviderChain.get_basic_credential_provider_chain()<br>credentials = chain.get_credentials()</pre><p><strong>Important Security Note:</strong> Never hardcode your AK and SK directly in your code. Store them in environment variables or encrypted configuration files and decrypt them when needed:</p><pre>import os<br>ak = os.environ[&quot;HUAWEICLOUD_SDK_AK&quot;]<br>sk = os.environ[&quot;HUAWEICLOUD_SDK_SK&quot;]</pre><h3>Client Initialization</h3><p>There are two main ways to initialize a service client in the Huawei Cloud SDK:</p><h3>Using Specified Endpoint</h3><pre>import os<br>from huaweicloudsdkcore.auth.credentials import BasicCredentials<br>from huaweicloudsdkvpc.v2 import VpcClient<br># Specify the endpoint<br>endpoint = &quot;https://vpc.cn-north-4.myhuaweicloud.com&quot;<br># Initialize credentials<br>ak = os.environ[&quot;HUAWEICLOUD_SDK_AK&quot;]<br>sk = os.environ[&quot;HUAWEICLOUD_SDK_SK&quot;]<br>project_id = &quot;your-project-id&quot;<br>credentials = BasicCredentials(ak, sk, project_id)<br># Initialize client<br>client = VpcClient.new_builder() \<br>    .with_credentials(credentials) \<br>    .with_endpoint(endpoint) \<br>    .build()</pre><h3>Using Specified Region (Recommended)</h3><pre>import os<br>from huaweicloudsdkcore.auth.credentials import BasicCredentials<br>from huaweicloudsdkvpc.v2 import VpcClient<br>from huaweicloudsdkvpc.v2.region.vpc_region import VpcRegion<br># Initialize credentials (project_id can be omitted when using regions)<br>ak = os.environ[&quot;HUAWEICLOUD_SDK_AK&quot;]<br>sk = os.environ[&quot;HUAWEICLOUD_SDK_SK&quot;]<br>credentials = BasicCredentials(ak, sk)<br># Initialize client with region<br>client = VpcClient.new_builder() \<br>    .with_credentials(credentials) \<br>    .with_region(VpcRegion.CN_NORTH_4) \<br>    .build()</pre><p>When using regions, the SDK will automatically query the project ID corresponding to the region, eliminating the need to provide it explicitly.</p><h3>Making Your First API Call</h3><p>Let’s make a simple API call to list VPCs:</p><pre>import os<br>from huaweicloudsdkcore.auth.credentials import BasicCredentials<br>from huaweicloudsdkcore.exceptions import exceptions<br>from huaweicloudsdkvpc.v2 import ListVpcsRequest, VpcClient<br>from huaweicloudsdkvpc.v2.region.vpc_region import VpcRegion<br># Load credentials from environment variables<br>ak = os.environ[&quot;HUAWEICLOUD_SDK_AK&quot;]<br>sk = os.environ[&quot;HUAWEICLOUD_SDK_SK&quot;]<br># Initialize credentials<br>credentials = BasicCredentials(ak, sk)<br># Initialize client<br>client = VpcClient.new_builder() \<br>    .with_credentials(credentials) \<br>    .with_region(VpcRegion.value_of(&quot;cn-north-4&quot;)) \<br>    .build()<br># Create and send request<br>try:<br>    request = ListVpcsRequest(limit=1)<br>    response = client.list_vpcs(request)<br>    print(response)<br>    <br>    # Access response object<br>    response_obj = response.to_json_object()<br>    print(response_obj[&quot;vpcs&quot;])<br>    <br>except exceptions.ClientRequestException as e:<br>    print(f&quot;Status code: {e.status_code}&quot;)<br>    print(f&quot;Request ID: {e.request_id}&quot;)<br>    print(f&quot;Error code: {e.error_code}&quot;)<br>    print(f&quot;Error message: {e.error_msg}&quot;)</pre><p>The structure is similar for other Huawei Cloud services; you just need to import the appropriate client and model classes.</p><h3>Working with HTTP Configuration</h3><p>The SDK provides various HTTP configuration options for your API requests:</p><h3>Default Configuration</h3><pre>config = HttpConfig.get_default_config()</pre><h3>Proxy Configuration</h3><pre>config.proxy_protocol = &#39;http&#39;<br>config.proxy_host = &#39;proxy.huaweicloud.com&#39;<br>config.proxy_port = 80<br>config.proxy_user = &#39;username&#39;<br>config.proxy_password = os.environ[&#39;PROXY_PASSWORD&#39;]</pre><h3>Timeout Configuration</h3><pre># Set unified timeout in seconds<br>config.timeout = 30<br># Or set specific timeouts<br>config.connect_timeout = 10<br>config.read_timeout = 30</pre><h3>SSL Configuration</h3><pre># Skip SSL verification (for testing only)<br>config.ignore_ssl_verification = True<br># Or configure CA certificate for SSL verification<br>config.ssl_ca_cert = &#39;/path/to/ca/certificate&#39;</pre><h3>Error Handling</h3><p>The SDK has a structured approach to error handling. Here are the main exception types:</p><p>Level 1DescriptionLevel 2DescriptionConnectionExceptionConnection exceptionHostUnreachableExceptionNetwork unreachable or access rejectedSslHandShakeExceptionSSL authentication is abnormalRequestTimeoutExceptionResponse timeout exceptionCallTimeoutExceptionServer fails to respond before timeoutRetryOutageExceptionNo valid response after max retriesServiceResponseExceptionServer response exceptionServerResponseExceptionInternal server error (500+)ClientRequestExceptionInvalid request parameter (400–499)</p><p>Example error handling:</p><pre>try:<br>    response = client.show_transcodings_template(request)<br>    print(response)<br>except exceptions.ConnectionException as e:<br>    print(f&quot;Connection error: {e}&quot;)<br>except exceptions.RequestTimeoutException as e:<br>    print(f&quot;Request timeout: {e}&quot;)<br>except exceptions.ServiceResponseException as e:<br>    print(f&quot;Status code: {e.status_code}&quot;)<br>    print(f&quot;Request ID: {e.request_id}&quot;)<br>    print(f&quot;Error code: {e.error_code}&quot;)<br>    print(f&quot;Error message: {e.error_msg}&quot;)</pre><h3>Asynchronous Operations</h3><p>For operations that might take time, the SDK supports asynchronous calls:</p><pre># Initialize asynchronous client<br>live_async_client = LiveAsyncClient.new_builder(LiveAsyncClient) \<br>    .with_http_config(config) \<br>    .with_credentials(credentials) \<br>    .with_endpoint(endpoint) \<br>    .build()<br># Send asynchronous request<br>request = model.ShowTranscodingsTemplateRequest(&quot;play.example.huaweicloud.com&quot;)<br>future_response = live_async_client.show_transcodings_template_async(request)<br># Get the response when ready<br>response = future_response.result()<br>print(response)</pre><h3>Logging</h3><p>The SDK provides built-in logging capabilities:</p><h3>File Logging</h3><pre>import logging<br>client = LiveAPIClient.new_builder(LiveAPIClient) \<br>    .with_http_config(config) \<br>    .with_credentials(credentials) \<br>    .with_endpoint(endpoint) \<br>    .with_file_log(path=&quot;sdk.log&quot;, log_level=logging.INFO) \<br>    .build()</pre><p>The with_file_log method supports these parameters:</p><ul><li>path: Log file path</li><li>log_level: Log level (default: INFO)</li><li>max_bytes: Maximum size of a log file (default: 10485760 bytes)</li><li>backup_count: Number of log files to keep (default: 5)</li></ul><h3>Console Logging</h3><pre>client = LiveAPIClient.new_builder(LiveAPIClient) \<br>    .with_http_config(config) \<br>    .with_credentials(credentials) \<br>    .with_endpoint(endpoint) \<br>    .with_stream_log(log_level=logging.INFO) \<br>    .build()</pre><p>The with_stream_log method supports:</p><ul><li>stream: Stream object (default: sys.stdout)</li><li>log_level: Log level (default: INFO)</li></ul><p>The log format is:</p><pre>&#39;%(asctime)s %(thread)d %(name)s %(filename)s %(lineno)d %(levelname)s %(message)s&#39;</pre><h3>Advanced Features</h3><h3>API Invoker</h3><h4>Custom Request Headers</h4><p>You can add custom headers to your requests:</p><pre>request = ListVpcsRequest(limit=1)<br>response = client.list_vpcs_invoker(request) \<br>    .add_header(&quot;X-Custom-Header&quot;, &quot;custom-value&quot;) \<br>    .add_header(&quot;X-Another-Header&quot;, &quot;another-value&quot;) \<br>    .invoke()</pre><p><strong>Note:</strong> Avoid specifying common headers like Host, Authorization, User-Agent, or Content-Type unless absolutely necessary, as this may cause errors.</p><h4>Retry Mechanism</h4><p>For handling transient failures, you can implement retry logic (available since v3.1.97):</p><pre>from huaweicloudsdkcore.exceptions.exceptions import ConnectionException<br>from huaweicloudsdkcore.retry.backoff_strategy import BackoffStrategies<br>request = ListVpcsRequest(limit=1)<br>response = client.list_vpcs_invoker(request).with_retry(<br>    # Retry on connection issues<br>    retry_condition=lambda resp, exc: isinstance(exc, ConnectionException),<br>    # Maximum 3 retries<br>    max_retries=3,<br>    # No delay between retries<br>    backoff_strategy=BackoffStrategies.NONE<br>).invoke()</pre><p>Available backoff strategies include:</p><ul><li>BackoffStrategies.NONE: No delay between retries</li><li>BackoffStrategies.EQUAL_JITTER: Equal jitter backoff</li><li>BackoffStrategies.FULL_JITTER: Full jitter exponential backoff</li><li>BackoffStrategies.EXPONENTIAL: Regular exponential backoff</li></ul><h3>File Upload and Download</h3><p>The SDK supports file operations. Here’s an example:</p><pre>from huaweicloudsdkcore.http.formdata import FormFile<br># Upload a file<br>image_file = FormFile(open(&quot;image.jpg&quot;, &quot;rb&quot;))<br>body = CreateImageWatermarkRequestBody(file=image_file, blind_watermark=&quot;watermark_text&quot;)<br>request.body = body<br>response = client.create_image_watermark(request)<br>image_file.close()<br># Download a file<br>def save(stream):<br>    with open(&quot;result.jpg&quot;, &quot;wb&quot;) as f:<br>        f.write(stream.content)<br>response.consume_download_stream(save)</pre><h3>HTTP Listeners for Debugging</h3><p>For debugging purposes, you can implement HTTP listeners to capture request and response details:</p><pre>def response_handler(**kwargs):<br>    response = kwargs.get(&quot;response&quot;)<br>    request = response.request<br><br>    info = &quot;&gt; Request %s %s HTTP/1.1&quot; % (request.method, request.path_url) + &quot;\n&quot;<br>    if len(request.headers) != 0:<br>        info = info + &quot;&gt; Headers:&quot; + &quot;\n&quot;<br>        for each in request.headers:<br>            info = info + &quot;    %s : %s&quot; % (each, request.headers[each]) + &quot;\n&quot;<br>    info = info + &quot;&gt; Body: %s&quot; % request.body + &quot;\n\n&quot;<br><br>    info = info + &quot;&lt; Response HTTP/1.1 %s &quot; % response.status_code + &quot;\n&quot;<br>    if len(response.headers) != 0:<br>        info = info + &quot;&lt; Headers:&quot; + &quot;\n&quot;<br>        for each in response.headers:<br>            info = info + &quot;    %s : %s&quot; % (each, response.headers[each],) + &quot;\n&quot;<br>    info = info + &quot;&lt; Body: %s&quot; % response.content<br>    print(info)<br><br>from huaweicloudsdkcore.http.http_handler import HttpHandler<br><br>client = VpcClient.new_builder() \<br>    .with_http_config(config) \<br>    .with_credentials(credentials) \<br>    .with_endpoint(endpoint) \<br>    .with_http_handler(HttpHandler().add_response_handler(response_handler)) \<br>    .build()</pre><p><strong>Warning:</strong> HTTP listeners expose sensitive information like headers and request bodies. Use them only for debugging and never in production environments.</p><h3>Best Practices</h3><ol><li><strong>Security</strong>: Never hardcode credentials in your code. Use environment variables or secure configuration management.</li><li><strong>Error Handling</strong>: Implement comprehensive error handling for all API calls, especially with proper logging.</li><li><strong>Connection Management</strong>: Configure appropriate timeouts to prevent hanging operations.</li><li><strong>Logging</strong>: In production, set the appropriate log level and ensure logs are properly rotated.</li><li><strong>SSL Verification</strong>: Always use SSL verification in production environments.</li><li><strong>Retry Logic</strong>: For critical operations, implement retry logic for transient failures.</li><li><strong>Asynchronous Operations</strong>: Use asynchronous calls for long-running operations to avoid blocking your application.</li><li><strong>Rate Limiting</strong>: Respect the service rate limits and implement backoff strategies if needed.</li></ol><h3>🔍 Conclusion</h3><p>🌟 The Huawei Cloud SDK for Python provides a powerful and flexible way to interact with Huawei Cloud services programmatically. With straightforward installation, multiple authentication methods, and comprehensive API coverage, it enables developers to automate their cloud infrastructure management efficiently.</p><p>Key takeaways from this guide:</p><ul><li>🔑 Multiple authentication methods are supported, including permanent AK/SK, temporary credentials, and federated authentication</li><li>🏗️ Client initialization can be done via specific endpoints or using region objects (recommended)</li><li>🛠️ HTTP configuration allows for customization of proxies, timeouts, and SSL settings</li><li>🔄 Flexible credential providers enable fetching authentication from environment variables, profiles, or instance metadata</li><li>🚦 Robust error handling hierarchies help manage different types of failures</li><li>⚡ Asynchronous operations support non-blocking API calls</li><li>🔁 Customizable retry mechanisms help handle transient failures gracefully</li><li>📝 Built-in logging capabilities provide visibility into SDK operations</li><li>🔒 Security best practices including environment-based credential management</li><li>📂 Support for file upload and download operations</li><li>🌐 Special configurations for cloud service alliance scenarios</li></ul><p>Whether you’re a DevOps engineer automating your infrastructure deployment, a developer integrating cloud services into your application, or a system administrator managing resources at scale, the Huawei Cloud SDK for Python streamlines your cloud management workflows and provides the flexibility needed for various use cases.</p><p>For easy debugging and exploration, Huawei Cloud offers <a href="https://apiexplorer.developer.intl.huaweicloud.com/apiexplorer/overview">API Explorer</a>, where you can retrieve API information, see SDK samples, and debug your code online.</p><p>As you continue your journey with Huawei Cloud, remember to regularly check their official documentation and <a href="https://github.com/huaweicloud/huaweicloud-sdk-python-v3/blob/master/CHANGELOG.md">CHANGELOG</a> for updates and new features. Cloud services evolve rapidly, and staying up-to-date with the latest SDK capabilities will ensure you’re making the most of what Huawei Cloud has to offer.</p><p>Happy coding in the cloud! ☁️🐍</p><h3>References</h3><ul><li><a href="https://github.com/huaweicloud/huaweicloud-sdk-python-v3">GitHub - huaweicloud/huaweicloud-sdk-python-v3</a></li><li><a href="https://support.huaweicloud.com/intl/en-us/sdk-python-devg-obs/obs_22_0001.html">Before You Start (SDK for Python)_Object Storage Service_Huawei Cloud</a></li><li><a href="https://support.huaweicloud.com/intl/tr-tr/ssdk-live/live_18_0003.html">Python SDK User Guide_Live_Huawei Cloud</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=727acc7668ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/huawei-developers/%EF%B8%8F-getting-started-with-huawei-cloud-sdk-core-for-python-a-comprehensive-guide-727acc7668ad">☁️ Getting Started with Huawei Cloud SDK Core for Python: A Comprehensive Guide🚀</a> was originally published in <a href="https://medium.com/huawei-developers">Huawei Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[☁️ API Gateway: 2025'te Modern Servis Mimarisi için Temel Taş]]></title>
            <link>https://medium.com/huawei-developers-tr/%EF%B8%8F-api-gateway-2025te-modern-servis-mimarisi-i%C3%A7in-temel-ta%C5%9F-4b7a42a2b0f4?source=rss-feb0d1d6abf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b7a42a2b0f4</guid>
            <category><![CDATA[huawei-cloud]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[huawei]]></category>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Berhan Ay]]></dc:creator>
            <pubDate>Fri, 14 Mar 2025 13:30:58 GMT</pubDate>
            <atom:updated>2025-03-14T13:30:58.514Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*V9fWOwda8Du-eMsI.jpg" /><figcaption>Article Banner</figcaption></figure><h3>🚀 Giriş</h3><p>Günümüzün birbirine bağlı dijital ortamında, Uygulama Programlama Arayüzleri (API’ler) farklı hizmetler ve uygulamalar arasında kesintisiz iletişimi sağlayan kritik bağlantılar olarak hizmet vermektedir. Bu değerli varlıkların ön cephesinde konumlandırılmış API Gateway, optimum performans, güvenlik ve ölçeklenebilirliği sağlamak için dikkatli bir şekilde uygulanmalıdır. Bu kapsamlı rehber, Huawei Cloud’un API Gateway (APIG) hizmetinin test edilmiş en iyi uygulamalarını ve uygulama stratejilerini inceleyerek, 2025 yılında API yönetim stratejilerini yükseltmek isteyen geliştiriciler ve mimarlar için pratik bilgiler sunmaktadır. 💡</p><h3>API Gateway Nedir?</h3><p>API Gateway (APIG), yeteneklerinizi paketlemek için herhangi bir ölçekte API’leri oluşturmanıza, yönetmenize ve dağıtmanıza olanak tanıyan bulut tabanlı bir ağ geçidi hizmetidir. Sadece birkaç tıklama ile iç sistemleri entegre edebilir, hizmet yeteneklerini paraya çevirebilir ve işlevleri minimum maliyet ve riskle seçici olarak sunabilirsiniz. APIG, kuruluşların hizmet yeteneklerini paraya çevirmelerine, AR-GE yatırımlarını azaltmalarına ve operasyonel verimliliği artırmak için temel kurumsal hizmetlere odaklanmalarına yardımcı olur.</p><h3>API Gateway’in Temel Özellikleri</h3><h3>1. API Yaşam Döngüsü Yönetimi</h3><p>Bir API’nin tam yaşam döngüsü, API’yi oluşturma, yayınlama, kaldırma ve silme işlemlerini içerir. APIG, tasarım, geliştirme, test etme, yayınlama ve operasyonlar ve bakım dahil olmak üzere tam yaşam döngüsü yönetimi sağlayarak, hizmet yeteneklerini hızlı ve verimli bir şekilde sunmanıza olanak tanır.</p><h3>2. Bulut Tabanlı Ağ Geçidi Entegrasyonu</h3><p>APIG, trafik girişini (Kubernetes Ingress) ve mikroservis yönetimini (Kubernetes Gateway API) tek bir ağ geçidinde entegre ederek, performansı artırır, mimarisi basitleştirir ve dağıtım ve O&amp;M maliyetlerini azaltır.</p><h3>3. Yerleşik Hata Ayıklama Araçları</h3><p>Yerleşik hata ayıklama aracı ile, farklı HTTP başlıkları ve istek gövdeleri kullanarak API’leri test edebilirsiniz. Bu araç, API geliştirme sürecini önemli ölçüde basitleştirir ve hem geliştirme hem de bakım maliyetlerini azaltır.</p><h3>4. Güçlü Sürüm Yönetimi</h3><p>API’ler farklı ortamlarda yayınlanabilir ve aynı ortamda yapılan her yeni yayın önceki sürümün yerine geçer. APIG, her API için sürüm detayları, açıklamalar, zaman damgaları ve ortam bilgileri dahil olmak üzere kapsamlı bir yayın geçmişi tutar. Bu, herhangi bir geçmiş sürüme geri dönmenize olanak tanıyarak hem karanlık yayın hem de sürüm yükseltme gereksinimlerini destekler.</p><h3>5. İncelikli İstek Kısıtlama</h3><p>APIG, sofistike istek kısıtlama yetenekleri sunar:</p><ul><li>API çağrı sıklığını kullanıcı, uygulama (kimlik bilgisi) veya IP adresine göre kontrol etme</li><li>API yolu, sorgu ve başlık parametreleri ile farklı istek kısıtlama limitleri yapılandırma</li><li>Kısıtlama doğruluğunu saniye, dakika, saat veya gün birimlerine ayarlama</li><li>Belirli kiracılar veya kimlik bilgileri için özel kısıtlama limitleri belirleme</li></ul><h3>6. Kapsamlı Güvenlik Özellikleri</h3><ul><li>TLS 1.1 ve TLS 1.2 ile doğrulanmış alan adı erişimi, mTLS iki yönlü kimlik doğrulama desteği ile</li><li>Belirli IP adreslerinden veya hesaplardan API erişimini sınırlamak için erişim kontrol politikaları</li><li>Anormal koşullar oluşursa arka uç hizmetlerini bozulma yoluyla korumak için devre kesici politikaları</li><li>Çoklu kimlik doğrulama mekanizmaları: AK/SK, işlev tabanlı özel yetkilendiriciler ve jetonlar</li></ul><h3>Uygulama Örnekleri</h3><h3>Örnek 1: CCE İş Yüklerini Güvenli Bir Şekilde Sunma</h3><p>Seçici olarak sunulması gereken birden fazla mikroservis içeren Cloud Container Engine (CCE) iş yükleri ile çalışırken, API Gateway zarif çözümler sunar:</p><p><strong>Yöntem 1 (Önerilen)</strong>: CCE iş yüklerindeki pod IP adreslerine erişmek için APIG’de bir yük dengeleme kanalı oluşturun:</p><pre># Uygulama adımları<br>1. CCE kümenizi hazırlayın (standart veya Turbo, VPC ağ modeli ile)<br>2. Konteyner CIDR bloğunu kaydedin ve adanmış ağ geçidinin rotalar alanına ekleyin<br>3. Uygun pod etiketleri (örn. app=deployment-demo, version=v1) ile CCE&#39;de bir iş yükü oluşturun<br>4. Şunlarla bir yük dengeleme kanalı kurun:<br>   - Konteyner port yapılandırması<br>   - Ağırlıklı round-robin (WRR) yönlendirme algoritması<br>   - Mikroservis yapılandırması (küme, ad alanı, iş yükü türü, servis etiketleri)<br>   - Sürüm yönetimi için sunucu grubu yapılandırması</pre><p>Gri ölçek yayınlamalar için (trafiği kademeli olarak v1&#39;den v2&#39;ye geçirme):</p><pre>1. Aynı app etiketi ama farklı sürüm etiketi (v2) ile yeni bir iş yükü oluşturun<br>2. Sunucu gruplarında trafik ağırlıklarını ayarlayın (örn. v1: %80, v2: %20)<br>3. Performans metriklerini izleyin ve trafiği kademeli olarak değiştirin<br>4. Migrasyon tamamlandıktan sonra eski sunucu grubunu kaldırın</pre><h3>Örnek 2: Özel Kimlik Doğrulama Uygulaması</h3><p>Standart kimlik doğrulama mekanizmaları iş ihtiyaçlarınız için yeterli olmadığında, özel kimlik doğrulama daha fazla esneklik sağlar:</p><pre># Temel kimlik doğrulama için örnek Python fonksiyonu<br>def handler(event, context):<br>    # İstek başlığında kimlik doğrulamayı doğrula<br>    if event[&quot;headers&quot;][&quot;authorization&quot;] == &#39;Basic dXNlcjE6cGFzc3dvcmQ=&#39;:<br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;body&#39;: json.dumps({<br>                &quot;status&quot;: &quot;allow&quot;,<br>                &quot;context&quot;: {<br>                    &quot;user_name&quot;: &quot;user1&quot;<br>                }<br>            })<br>        }<br>    else:<br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;body&#39;: json.dumps({<br>                &quot;status&quot;: &quot;deny&quot;,<br>                &quot;context&quot;: {<br>                    &quot;code&quot;: &quot;1001&quot;,<br>                    &quot;message&quot;: &quot;yanlış kullanıcı adı veya şifre&quot;,<br>                    &quot;authorizer_success&quot;: &quot;false&quot;<br>                }<br>            })<br>        }</pre><p>Uygulama adımları:</p><ol><li>Özel başlıkları veya sorgu parametrelerini doğrulayan bir fonksiyon oluşturun</li><li>Fonksiyon içinde kimlik doğrulama mantığınızı uygulayın</li><li>Fonksiyonu özel yetkilendirici olarak APIG’ye bağlayın</li><li>API’leri bu özel yetkilendiriciyi kullanacak şekilde yapılandırın</li><li>Kimlik doğrulama başarısızlıkları için uygun yanıt şablonları oluşturun</li></ol><h3>Örnek 3: VPC Arası Hizmet Sunumu</h3><p>API Gateway’iniz ve arka uç hizmetleriniz farklı VPC’lerde bulunduğunda, VPC arası bağlantı kurmak önemli hale gelir:</p><pre># Uygulama adımları<br>1. Ağ geçidi ve arka uç hizmetleri için ayrı VPC&#39;ler oluşturun<br>2. İlk VPC&#39;de adanmış bir ağ geçidi oluşturun<br>3. İkinci VPC&#39;de bir yük dengeleyici kurun<br>4. İletişimi etkinleştirmek için VPC eşleştirme bağlantısı yapılandırın<br>5. VPC&#39;ler arasında trafik akışını sağlamak için gerekli rotaları ekleyin<br>6. Yük dengeleyicinin IP adresini işaret eden API&#39;ler oluşturun</pre><p>Bu mimari, ağ geçidiniz ve arka uç hizmetleriniz arasında mantıksal bir ayrım sağlarken kesintisiz iletişimi garanti eder.</p><h3>Örnek 4: Gelişmiş İstek Kısıtlama Uygulaması</h3><p>Sofistike trafik yönetimi gereksinimleri için, İstek Kısıtlama 2.0 incelikli kontrol sağlar:</p><pre>{<br>  &quot;name&quot;: &quot;gelismis-kisitlama-politikasi&quot;,<br>  &quot;throttling&quot;: &quot;Yüksek-performans&quot;,<br>  &quot;policyType&quot;: &quot;API-specific&quot;,<br>  &quot;period&quot;: &quot;60s&quot;,<br>  &quot;maxAPIRequests&quot;: 10,<br>  &quot;maxUserRequests&quot;: 5,<br>  &quot;parameterBasedThrottling&quot;: [<br>    {<br>      &quot;parameter&quot;: &quot;Host&quot;,<br>      &quot;location&quot;: &quot;header&quot;,<br>      &quot;rules&quot;: [<br>        {<br>          &quot;conditions&quot;: [{&quot;equals&quot;: &quot;www.example.com&quot;}],<br>          &quot;maxRequests&quot;: 10,<br>          &quot;period&quot;: &quot;60s&quot;<br>        }<br>      ]<br>    }<br>  ],<br>  &quot;excludedThrottling&quot;: {<br>    &quot;credentials&quot;: [&quot;app1-credential-id&quot;, &quot;app2-credential-id&quot;],<br>    &quot;maxRequests&quot;: 20<br>  }<br>}</pre><p>Uygulama adımları:</p><ol><li>İstek kısıtlama 2.0 politikası oluşturun</li><li>Temel kısıtlamayı yapılandırın (örn. API başına dakikada 10 istek)</li><li>Daha ayrıntılı kontrol için parametre tabanlı kurallar ekleyin</li><li>Özel istemciler için hariç tutulan kısıtlamayı yapılandırın</li><li>Politikayı uygun API’lere bağlayın</li></ol><h3>Örnek 5: API Gateway’i WAF ile Koruma</h3><p>API Gateway’inizi ve arka uç hizmetlerinizi kötü amaçlı saldırılardan korumak için Web Uygulama Güvenlik Duvarını (WAF) koruyucu bir katman olarak dağıtın:</p><pre># Yöntem 1: API Grubu Hata Ayıklama Alanını WAF&#39;a Kaydetme<br>1. Bir API grubu oluşturun ve hata ayıklama alan adını not edin<br>2. Bu alanı uygun sertifika yapılandırması ile WAF&#39;a ekleyin<br>3. Alan adını APIG&#39;deki API grubuna bağlayın<br>4. İstemci IP adreslerini korumak için real_ip_from_xff&#39;i etkinleştirin</pre><pre># Yöntem 2: Ağ Geçidi Giriş Erişim Adresini Kullanma<br>1. Ağ geçidinin giriş erişim adresini alın<br>2. DEFAULT grubunda API&#39;ler oluşturun<br>3. WAF&#39;ı ağ geçidinin giriş adresi ile yapılandırın<br>4. Özel alanı DEFAULT grubuna bağlayın<br>5. İstemci IP adreslerini korumak için real_ip_from_xff&#39;i etkinleştirin</pre><p>Her iki yaklaşım da tam işlevselliği korurken API’lerinizi yaygın web saldırılarından etkili bir şekilde korur.</p><h3>API Gateway Spesifikasyonları ve Performans</h3><p>APIG, çeşitli iş yükü gereksinimlerini karşılamak için farklı sürümler sunar:</p><ul><li><strong>Temel Sürüm</strong>: Geliştirme ve test ortamları için uygun, 1.600 QPS’ye kadar sunar</li><li><strong>Profesyonel Sürüm</strong>: 3.600 QPS’ye kadar küçük üretim iş yükleri için uygun</li><li><strong>Kurumsal Sürüm</strong>: 9.000 QPS’ye kadar orta iş yükleri için tasarlanmıştır</li><li><strong>Platinum Sürüm</strong>: 55.000 QPS’ye kadar yüksek verimli uygulamalar için optimize edilmiştir</li><li><strong>Platinum 2 Sürüm</strong>: 72.000 QPS’ye kadar aşırı performans için tasarlanmıştır</li></ul><p>Verim, yanıt boyutu, HTTPS kullanımı ve gzip sıkıştırması gibi faktörlere bağlı olarak değişir. Optimum performans için, ağ geçidi sürümünüzü seçerken bu spesifikasyonları göz önünde bulundurun.</p><h3>API Gateway Uygulaması için En İyi Uygulamalar</h3><p><strong>1. Net sürüm stratejileri ile API’ler tasarlayın</strong></p><ul><li>Semantik sürümlendirme ilkelerini kullanın</li><li>Farklı aşamalar için ortam değişkenleri uygulayın</li><li>Net kullanımdan kaldırma politikaları oluşturun</li></ul><p><strong>2. Kapsamlı güvenlik önlemleri uygulayın</strong></p><ul><li>Hassas işlemler için karşılıklı TLS kullanın</li><li>Farklı API türleri için uygun kimlik doğrulama uygulayın</li><li>Erişim kontrolünü istek kısıtlama ile birleştirin</li></ul><p><strong>3. Önbelleğe alma yoluyla performansı optimize edin</strong></p><ul><li>Sık erişilen veriler için yanıt önbelleğe alma uygulayın</li><li>Uygun önbellek geçersiz kılma stratejileri yapılandırın</li><li>Önbellek isabet oranlarını izleyin</li></ul><p><strong>4. Güçlü izleme ve uyarı sistemleri kurun</strong></p><ul><li>Temel metrikleri takip edin: istek sayıları, gecikme, hata oranları</li><li>Anormal desenler için uyarılar ayarlayın</li><li>Detaylı sorun giderme için günlük kaydı kullanın</li></ul><p><strong>5. Felaket kurtarma için plan yapın</strong></p><ul><li>Yüksek kullanılabilirlik için çoklu AZ dağıtımları uygulayın</li><li>Net yedekleme ve geri yükleme prosedürleri oluşturun</li><li>Düzenli yük devretme testleri yapın</li></ul><h3>Paylaşımlı Gateway’den Adanmış Gateway’e Geçiş</h3><p>Huawei Cloud paylaşımlı ağ geçitlerini aşamalı olarak kaldırdığından, adanmış ağ geçitlerine geçiş önemli hale gelir:</p><p><strong>1. Geçiş öncesi kontroller yapın</strong>:</p><ul><li>KooGallery kullanımını doğrulayın</li><li>Kullanıcılar arası yetkilendirmeyi kontrol edin</li><li>VPC kanal türlerini gözden geçirin</li><li>DataArts tarafından oluşturulan kaynakları belirleyin</li></ul><p><strong>2. İş yükünüze göre uygun spesifikasyonlara sahip adanmış bir ağ geçidi oluşturun</strong></p><p><strong>3. Teknik destek aracılığıyla migrasyon için başvurun</strong></p><p><strong>4. Doğrulayın ve trafiği değiştirin</strong>:</p><ul><li>Adanmış ağ geçidinde API’leri test edin</li><li>DNS kayıtlarını yeni ağ geçidini işaret edecek şekilde güncelleyin</li><li>Gerekirse VPC arası bağlantıyı yapılandırın</li></ul><p>Potansiyel etkilere dikkat edin:</p><ul><li>Değiştirilen genel ve özel IP adresleri</li><li>Değiştirilmiş hata ayıklama alan adları</li><li>Ağ yapılandırma ayarlamaları</li></ul><h3>🌟 Sonuç</h3><p>Bu API Gateway en iyi uygulamalarını uygulamak, API ekosisteminizin güvenliğini, performansını ve yönetilebilirliğini önemli ölçüde artıracaktır. Servisleri seçici olarak sunarak, güçlü kimlik doğrulama uygulayarak, ayrıntılı kısıtlama politikaları uygulayarak, güvenlik katmanları dağıtarak ve göçleri dikkatle planlayarak, 2025 ve sonrasındaki dijital hizmetleriniz için sağlam bir temel olarak hizmet edecek esnek, ölçeklenebilir bir API mimarisi oluşturacaksınız. 🔒</p><p>İster konteynere alınmış iş yüklerini yönetiyor, ister çok faktörlü kimlik doğrulama uyguluyor, ister trafik akışını optimize ediyor, ister API uç noktalarınızı güvence altına alıyor olun, Huawei Cloud’un API Gateway’i, API yönetim stratejinizi dönüştürmek için gereken kapsamlı araçları ve yetenekleri sağlar. ✨</p><h3>Referanslar</h3><p><a href="https://support.huaweicloud.com/intl/en-us/api-gateway/">https://support.huaweicloud.com/intl/en-us/api-gateway/</a></p><p>Fowler, M. (2023). “Patterns of Enterprise API Management: The Gateway Pattern.” O’Reilly Media.</p><p>Richardson, L. (2024). “Microservices Patterns: Building Effective API Gateways.” Manning Publications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b7a42a2b0f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/huawei-developers-tr/%EF%B8%8F-api-gateway-2025te-modern-servis-mimarisi-i%C3%A7in-temel-ta%C5%9F-4b7a42a2b0f4">☁️ API Gateway: 2025&#39;te Modern Servis Mimarisi için Temel Taş🚀</a> was originally published in <a href="https://medium.com/huawei-developers-tr">Huawei Developers - Türkiye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Huawei Cloud DLI ile PostgreSQL’den GaussDB(DWS)’e Gerçek Zamanlı Veri Aktarımı & İşleme]]></title>
            <link>https://medium.com/huawei-developers-tr/huawei-cloud-dli-ile-postgresqlden-gaussdb-dws-e-ger%C3%A7ek-zamanl%C4%B1-veri-aktar%C4%B1m%C4%B1-i%CC%87%C5%9Fleme-cc678487b1ab?source=rss-feb0d1d6abf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc678487b1ab</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[huawei-cloud]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[flink]]></category>
            <dc:creator><![CDATA[Berhan Ay]]></dc:creator>
            <pubDate>Wed, 05 Mar 2025 16:21:45 GMT</pubDate>
            <atom:updated>2025-03-05T16:21:45.833Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Zbuw7vyr4367irn.jpg" /><figcaption>Cover Photo</figcaption></figure><p>🔍 <strong>Veri entegrasyonu</strong>, günümüzün hızlı veri işleme dünyasında oldukça kritik bir görevdir. 🚀 Büyük veri kümesi ile çalışan organizasyonlar için bu verileri <strong>depolamak, işlemek ve analiz etmek</strong> 🛠️ hiç olmadığı kadar önemli hale geldi.</p><p>🌐 <strong>Huawei Cloud</strong>, verilerinizi etkili bir şekilde yönetebileceğiniz güçlü bir platform olan <strong>Data Lake Insight (DLI)</strong> sunuyor. 💡 Bu makalede, <strong>DLI</strong> kullanarak nasıl <strong>veri akışı oluşturabileceğinizi</strong> ve <strong>PostgreSQL veritabanından GaussDB(DWS) veritabanına veri aktarımını</strong> 📊 nasıl gerçekleştirebileceğinizi adım adım göstereceğim. ✅</p><h3>1. Data Lake Insight Nedir?</h3><p>Huawei Cloud’un <strong>Data Lake Insight (DLI)</strong> platformu, büyük veri kümelerini depolamak ve analiz etmek için geliştirilmiş güçlü bir bulut servisidir. DLI, verilerinizi <strong>SQL</strong> ve <strong>Flink</strong> gibi işleme motorları kullanarak hızlı ve verimli bir şekilde işlemeyi mümkün kılar. Ayrıca, DLI üzerinden veritabanları arasındaki veri akışını yönlendirebilir, verilerinizi gerçek zamanlı olarak takip edebilir ve analiz edebilirsiniz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iiJIk1nGyf3X_uL0eEBU6w.png" /><figcaption><strong>Screenshot 1</strong>: DLI Konsolu Görünümü</figcaption></figure><h3>2. Queue Oluşturma</h3><p>İlk adımımız, <strong>Queue</strong> yani kuyruk oluşturmaktır. DLI platformu üzerinde verilerinizi işlerken, veritabanı bağlantıları ve işlem kaynakları için bir <strong>queue</strong> (kuyruk) oluşturmanız gerekir. Bu kuyruk, verileri işlemek ve yönetmek için DLI platformuna bağlanmanızı sağlayacak.</p><p>Queue oluşturmak için DLI konsoluna giriş yapıyoruz ve sol menüden <strong>Resources &gt; Queue Management</strong> kısmına tıklıyoruz. Burada <strong>Buy Queue</strong> seçeneğini seçiyoruz. Ardından, kuyruk ismini belirliyor, bölge ve proje ayarlarını varsayılan bırakıyoruz. Queue’nun türünü <strong>For general purpose</strong> olarak seçip, <strong>Dedicated Resource Mode</strong>’u aktif hale getiriyoruz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1fQIqi1AhryPTor5eiL-Sw.png" /><figcaption><strong>Screenshot 2</strong>: Queue Oluşturma Ekranı</figcaption></figure><h3>3. RDS PostgreSQL Veritabanı ve Tablosu Oluşturma</h3><p>Bir sonraki adımda, <strong>RDS PostgreSQL</strong> veritabanımıza bağlanacağız. Huawei Cloud üzerinde <strong>Relational Database Service (RDS)</strong>, PostgreSQL veritabanlarını barındırmanızı sağlar. Burada veritabanımızı ve tablomuzu oluşturacağız.</p><p>İlk olarak, RDS konsoluna giriş yapıyoruz ve hedef PostgreSQL DB instance’ına tıklıyoruz. Ardından, veritabanı için bir şema ve tablo oluşturuyoruz. Örneğin:</p><pre>create table test.cdc_order(<br>  order_id VARCHAR,<br>  order_channel VARCHAR,<br>  order_time VARCHAR,<br>  pay_amount FLOAT8,<br>  real_pay FLOAT8,<br>  pay_time VARCHAR,<br>  user_id VARCHAR,<br>  user_name VARCHAR,<br>  area_id VARCHAR,<br>  primary key(order_id));</pre><p>Ayrıca, bu tablonun <strong>REPLICA IDENTITY FULL</strong> olarak ayarlanması gerekmektedir. Bu, <strong>Change Data Capture (CDC)</strong> özelliğinin aktif olması için gereklidir.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RugArUUf5GHtJQjqtXKc8g.png" /><figcaption><strong>Screenshot 3.1</strong>: RDS PostgreSQL Cluster’ı oluşutrma</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8e5EL-jT2RRl9y2nLzttA.png" /><figcaption><strong>Screenshot 3.2</strong>: RDS PostgreSQL Tablo Oluşturma</figcaption></figure><h3>4. GaussDB(DWS) Veritabanı ve Tablosu Oluşturma</h3><p>Veri aktarımının yapılacağı hedef veritabanı <strong>GaussDB(DWS)</strong> olacaktır. GaussDB, Huawei Cloud’un veri ambarı çözümüdür ve büyük veri analizleri için optimize edilmiştir. Şimdi, GaussDB(DWS) üzerinde bir veritabanı ve tablo oluşturacağız.</p><p>Öncelikle <strong>GaussDB(DWS) konsoluna</strong> giriş yapıyoruz ve yeni bir veritabanı oluşturuyoruz. Bu veritabanını <strong>testdwsdb</strong> olarak adlandırıyoruz. Ardından, <strong>gsql CLI</strong> aracı ile bağlanıp gerekli tablomuzu oluşturuyoruz:</p><pre>create schema test;<br>set current_schema= test;<br>create table dws_order (<br>  order_id VARCHAR,<br>  order_channel VARCHAR,<br>  order_time VARCHAR,<br>  pay_amount FLOAT8,<br>  real_pay FLOAT8,<br>  pay_time VARCHAR,<br>  user_id VARCHAR,<br>  user_name VARCHAR,<br>  area_id VARCHAR<br>);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XUBalnxTJ-ccXxs6MBu1ZQ.png" /><figcaption><strong>Screenshot 4.1</strong>: GaussDB(DWS) Cluster Oluşturma</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a64wqdKgC6gB8oVZnqxwkg.png" /><figcaption><strong>Screenshot 4.2</strong>: GaussDB(DWS) Bağlantısı ve Tablo Oluşturma</figcaption></figure><h3>5. Veritabanları Arasında Bağlantı Kurma</h3><p>Artık her iki veritabanı arasında bağlantı kurmamız gerekiyor. Bunun için <strong>Enhanced Datasource Connection</strong> özelliğini kullanacağız. Bu adımda, DLI ile hem RDS PostgreSQL hem de GaussDB(DWS) arasında bağlantılar oluşturacağız.</p><p>İlk olarak, <strong>RDS</strong> bağlantısını kuracağız. Bağlantı için gerekli bilgileri (IP adresi, port numarası, VPC, subnet) aldıktan sonra, DLI konsolunda <strong>Datasource Connections</strong> kısmına gidip yeni bir <strong>enhanced connection</strong> oluşturacağız. Burada, <strong>RDS</strong> ve <strong>GaussDB(DWS)</strong> instance’larının bağlantı bilgilerini doğru şekilde girmeniz gerekiyor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLRe4IxPd_myRojm_7xaEQ.png" /><figcaption><strong>Screenshot 5</strong>: Datasource Bağlantıları</figcaption></figure><p>6. Flink Job’u Çalıştırma</p><p>Şimdi veri işleme sürecine geçiyoruz. <strong>Flink</strong>, büyük veri işlemleri için kullanılan açık kaynaklı bir işleme motorudur. DLI üzerinden <strong>Flink SQL</strong> kullanarak veritabanındaki verileri işleyebilir ve sonuçları hedef veritabanımıza aktarabiliriz.</p><p>DLI konsolunda <strong>Job Management &gt; Flink Jobs</strong> kısmına giderek yeni bir job oluşturuyoruz. Bu job, <strong>PostgreSQL CDC</strong> verilerini alıp, belirlediğimiz <strong>GaussDB(DWS)</strong> veritabanına aktarmak için çalışacak.</p><p>Aşağıdaki SQL komutlarını kullanarak verilerin aktarımını gerçekleştireceğiz:</p><pre>create table PostgreCdcSource(<br>  order_id string,<br>  order_channel string,<br>  order_time string,<br>  pay_amount double,<br>  real_pay double,<br>  pay_time string,<br>  user_id string,<br>  user_name string,<br>  area_id STRING,<br>  primary key (order_id) not enforced<br>) with (<br>  &#39;connector&#39; = &#39;postgres-cdc&#39;,<br>  &#39;hostname&#39; = &#39;ip adresi&#39;,<br>  &#39;port&#39;= &#39;5432&#39;,<br>  &#39;database-name&#39; = &#39;testrdsdb&#39;,<br>  &#39;schema-name&#39; = &#39;test&#39;,<br>  &#39;table-name&#39; = &#39;cdc_order&#39;<br>);</pre><pre>create table dwsSink(<br>  order_id string,<br>  order_channel string,<br>  order_time string,<br>  pay_amount double,<br>  real_pay double,<br>  pay_time string,<br>  user_id string,<br>  user_name string,<br>  area_id STRING,<br>  primary key(order_id) not enforced<br>) with (<br>  &#39;connector&#39; = &#39;gaussdb&#39;,<br>  &#39;url&#39;=&#39;jdbc:gaussdb://ip adresi:port/testdwsdb&#39;,<br>  &#39;table-name&#39; = &#39;test.dws_order&#39;,<br>  &#39;username&#39; = &#39;xxxxx&#39;,<br>  &#39;password&#39; = &#39;xxxxx&#39;,<br>  &#39;write.mode&#39; = &#39;insert&#39;<br>);</pre><pre>insert into dwsSink select * from PostgreCdcSource where pay_amount &gt; 100;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fZGjcGpPJYAQOiT81oS5w.png" /><figcaption><strong>Screenshot 6.1</strong>: Agency Permission’ı Oluşturmak</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zUuKuBk52G3ggKemjGczrg.png" /><figcaption><strong>Screenshot 6.2</strong>: Flink SQL Job’unun Tanımlanması</figcaption></figure><h3>7. Sonuçları Kontrol Etme</h3><p>Flink job’unu başlattıktan sonra, verilerin doğru şekilde aktarıldığından emin olmak için her iki veritabanını da sorgulamalıyız. RDS PostgreSQL veritabanına veri ekledikten sonra, GaussDB(DWS) üzerinde ilgili tabloyu sorgulayarak verinin aktarıldığını kontrol edebilirsiniz:</p><pre>select * from test.dws_order;</pre><h3>Alternatif Olarak MRS ile Veri İşleme</h3><p>Yukarıdaki işlemleri <strong>Data Lake Insight (DLI)</strong> platformu üzerinden gerçekleştirdik, ancak benzer işlemleri <strong>MRS (MapReduce Service)</strong> servisi kullanarak da gerçekleştirebilirsiniz. MRS, büyük veri işlemleri ve analizleri için başka bir güçlü çözüm sunar. <strong>Flink</strong> ve <strong>Spark</strong> gibi açık kaynaklı işleme motorları MRS içinde de yer alır. Bu motorlar, DLI’ye benzer şekilde veri akışlarını yönetmenize, veri setlerinizi işleyip analiz etmenize olanak tanır.</p><p>MRS servisinde, <strong>Flink</strong> veya <strong>Spark</strong> kullanarak verileri gerçek zamanlı olarak işleyebilir ve <strong>PostgreSQL</strong> ile <strong>GaussDB(DWS)</strong> arasında veri entegrasyonu sağlayabilirsiniz. Bu seçenek, daha geniş veri işleme ve analiz yetenekleri sunar ve verilerinizi daha esnek bir şekilde yönetmenize yardımcı olur.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cd1BBZDzDwx5kzNibSJpSQ.png" /><figcaption><strong>Screenshot 8</strong>: MRS Arayüzü</figcaption></figure><p>Sonuç olarak, DLI ve MRS gibi servislerle büyük veri yönetimini ve işleme süreçlerinizi kolayca kontrol edebilirsiniz. Hangi servisi kullanacağınız, organizasyonunuzun ihtiyaçlarına ve altyapınıza bağlıdır.</p><ul><li><a href="https://www.huaweicloud.com/intl/tr-tr/product/dli.html">Data Lake Insight (DLI) - Büyük Veri Analizi | Huawei Cloud</a></li><li><a href="https://carrier.huawei.com/en/products/it-new/cloud/datapool">Huawei Cloud Intelligent Data Lake</a></li><li><a href="https://e.huawei.com/fr-MC/solutions/cloud-computing/big-data">FusionInsight Intelligent Data Lake - Huawei Enterprise</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc678487b1ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/huawei-developers-tr/huawei-cloud-dli-ile-postgresqlden-gaussdb-dws-e-ger%C3%A7ek-zamanl%C4%B1-veri-aktar%C4%B1m%C4%B1-i%CC%87%C5%9Fleme-cc678487b1ab">Huawei Cloud DLI ile PostgreSQL’den GaussDB(DWS)’e Gerçek Zamanlı Veri Aktarımı &amp; İşleme🚀🔄📊</a> was originally published in <a href="https://medium.com/huawei-developers-tr">Huawei Developers - Türkiye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[☁️ Build, Manage, and Analyze: The Ultimate Guide to Huawei Cloud MRS Platform]]></title>
            <link>https://medium.com/huawei-developers/%EF%B8%8F-build-manage-and-analyze-the-ultimate-guide-to-huawei-cloud-mrs-platform-bb9bce72a2cd?source=rss-feb0d1d6abf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb9bce72a2cd</guid>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[huawei-cloud]]></category>
            <category><![CDATA[mapreduce]]></category>
            <category><![CDATA[hdfs]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Berhan Ay]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 15:31:06 GMT</pubDate>
            <atom:updated>2025-01-16T15:31:27.057Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cRRB4C6vaNsFmqPc64lU5g.jpeg" /><figcaption>Cover of Article</figcaption></figure><p>Welcome to this <strong>step-by-step guide</strong> on mastering cluster configuration and data analysis using Huawei Cloud’s MRS (MapReduce Service) platform! Whether you’re a beginner exploring big data or a seasoned professional refining your skills, this guide has something for everyone. 📊</p><p>We’ll walk you through the essentials:</p><ul><li>🛠️ <strong>Creating and Configuring Clusters</strong></li><li>🔒 <strong>Setting Up Security</strong></li><li>🗂️ <strong>Managing Data with Hive</strong></li><li>📈 <strong>Performing Hadoop-Based Data Analysis</strong></li></ul><p>By the end, you’ll be fully equipped to utilize Huawei Cloud’s MRS platform for scalable and efficient big data solutions. Let’s dive in and unlock the power of cloud-based analytics! 💡✨</p><h3>Procedure Overview:</h3><h3>Step 1: Create a Cluster 🚀</h3><h4>1. Accessing the Cluster Creation Page 🖥️</h4><p><strong>What is the MRS Normal Version?</strong><br>💡 The <strong>MRS normal version</strong> supports basic cluster operations like configuration, management, and O&amp;M. The <strong>Long-Term Support (LTS)</strong> version also includes features such as version upgrades and multi-AZ deployment for enhanced flexibility.</p><p><strong>What is a Hadoop Analysis Cluster?</strong><br>📊 A <strong>Hadoop analysis cluster</strong> leverages open-source Hadoop ecosystem components for data analysis. Key components include:</p><ul><li>🛠️ <strong>Yarn</strong>: Manages cluster resources</li><li>📂 <strong>Hive</strong> &amp; <strong>Spark</strong>: Handles large-scale data storage and computing</li><li>🔄 <strong>Spark Streaming</strong> &amp; <strong>Flink</strong>: Provides streaming data computation</li><li>🔍 <strong>Presto</strong> &amp; <strong>Tez</strong>: Enables interactive queries and distributed computing</li></ul><p><strong>What is Kerberos Authentication?</strong><br>🔒 <strong>Kerberos</strong> is a security protocol that uses symmetric key management to authenticate clients and servers. It ensures:</p><ul><li>🔑 <strong>Data Integrity</strong></li><li>🔐 Protection from eavesdropping and replay attacks</li></ul><p>By combining these tools and features, you can configure secure, scalable clusters to handle your data analytics tasks seamlessly. 🚀</p><p>Log in to the Huawei Cloud console. Choose Service List &gt; Analytics &gt; MapReduce Service to access the MRS console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*iKEJ9-5Dx777Ihwh.png" /><figcaption>MapReduce Service</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*2AhXeaXBJ53SXxo1.png" /><figcaption>Active Cluster</figcaption></figure><h4>2. Creating the Cluster</h4><ul><li><strong>Navigate to</strong>: Huawei Cloud Console &gt; Service List &gt; Analytics &gt; MapReduce Service (MRS).</li><li>Click <strong>Buy Cluster</strong> on the top right, and configure the following parameters:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*7uQ8C40P9cI4S2NK.png" /><figcaption>Buy Cluster</figcaption></figure><p>Parameters:</p><ul><li><strong>Region</strong>: AP-Singapore</li><li><strong>Billing Mode</strong>: Pay-per-use</li><li><strong>Cluster Name</strong>: mrs_test</li><li><strong>Version Type</strong>: Normal</li><li><strong>Cluster Version</strong>: MRS 3.1.0</li><li><strong>Component</strong>: Hadoop Analysis Cluster</li><li><strong>AZ</strong>: Retain the default value</li><li><strong>VPC</strong>: vpc-mrs</li><li><strong>Subnet</strong>: subnet_mrs</li><li><strong>Cluster Node</strong>: Set <strong>Node Count</strong> to <strong>2</strong> for the master node group and <strong>3</strong> for the core node group</li><li><strong>Kerberos Authentication</strong>: Do not enable this function</li><li><strong>Username</strong>: root/admin</li><li><strong>Password</strong>: Mrstest@123</li><li><strong>Secure Communications</strong>: Select <strong>Enable</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*mvuXlMRapqPDLgTK.png" /><figcaption>Configuration 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*aqidneRDY4XSqcty.png" /><figcaption>Configuration 2</figcaption></figure><ul><li>After entering all the necessary parameters, click <strong>Buy Now</strong>. You can view the created cluster in the cluster list.</li></ul><h3>Step 2: Bind an EIP to the Created Cluster 🌐</h3><h4>What is an EIP? 💡</h4><p>An <strong>EIP (Elastic IP)</strong> is a public, independent IP address that provides:</p><ul><li>🌍 <strong>Public Network Access</strong>: Bind or unbind it to/from ECSs, BMSs, virtual IPs, load balancers, and NAT gateways.</li><li>📶 <strong>Flexible Bandwidth Management</strong>: Allocate bandwidth as needed for optimized performance.</li></ul><p>By binding an EIP, you enable your cluster to communicate with the public network, ensuring seamless data transfer and access. 🚀</p><ol><li><strong>Access the ECS Console<br>Navigate to</strong>: Service List &gt; Compute &gt; Elastic Cloud Server (ECS) to view the five ECSs created in the cluster. Find the active ECS <strong>…master1</strong>.</li><li><strong>Bind the EIP</strong></li></ol><ul><li>Click <strong>More</strong>, choose <strong>Manage Network &gt; Bind EIP</strong>, and bind a preset EIP to the ECS.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*OH1Cyh7ZEUvQAkOf.png" /><figcaption>Binding EIP 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*ojB2myaygFKzmUcW.png" /><figcaption>Binding EIP 2</figcaption></figure><ul><li>Select an EIP with bandwidth of 100 Mbit/s.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/0*mNN9JrFiIF55pFj8.png" /><figcaption>Select EIP</figcaption></figure><ul><li>Once binding is complete, you can view the EIP bound to the active ECS in the ECS list.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*rAABRQTSIbnqdU5y.png" /><figcaption>Check EIP</figcaption></figure><h3>Step 3: Configure Security Group Rules 🔒</h3><h4>What is a Security Group Rule? 💡</h4><p>A <strong>security group rule</strong> defines the allowed traffic flow to and from ECSs (Elastic Cloud Servers) within a security group. These rules ensure:</p><ul><li>📥 Controlled <strong>Inbound Traffic</strong></li><li>📤 Managed <strong>Outbound Traffic</strong></li></ul><p>Security group rules are critical for maintaining the security and proper functioning of your cluster.</p><h4>What are the Differences Between Inbound and Outbound Security Group Rules? ⚙️</h4><ul><li><strong>Inbound Rules</strong>:<br>📥 Allow data packets <strong>from other instances</strong> in the same security group.</li><li><strong>Outbound Rules</strong>:<br>📤 Allow data packets <strong>to other instances</strong> in the same security group.</li></ul><p>These rules ensure that traffic flows securely and only as intended within your cluster’s environment. 🚀</p><p><strong>1. Access the Cluster Details</strong></p><ul><li>Return to the MRS cluster list page and click the cluster name to access the cluster details page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*d5bM1MNBo3GBlOjn.png" /><figcaption>Acces Cluster</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*8Xp-crqDaWm_0ZSn.png" /><figcaption>Synchronize Cluster</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*ZKS4S9RH7sA6Im-6.png" /><figcaption>Check the Status</figcaption></figure><p><strong>2. Bind EIP and Security Group</strong></p><ul><li>Click <strong>Access Manager</strong> to bind an EIP and security group to the cluster.</li><li>After the binding is successful, the Manager page will be displayed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*lnESi6pMmIH2Hn8d.png" /><figcaption>Access Manager</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/0*26HEbVgwrML79hgc.png" /><figcaption>Access MRS Manager</figcaption></figure><p><strong>3. Configure Inbound Security Rules</strong></p><ul><li>Click <strong>Manage Security Group Rule</strong> next to <strong>Add Security Group Rule</strong>.</li><li>Choose <strong>Inbound Rules</strong> then click <strong>Add Rule</strong>. Enable ports <strong>22</strong> and <strong>9022</strong> for remote login.</li><li><strong>Note</strong>: Using the network segment 0.0.0.0/0 is only for exercises. Avoid using it in production environments as it allows all requests.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*PwLDzzVmNFRuA2Fg.png" /><figcaption>Manage the Security Group</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*nGVEyInk4icruO5g.png" /><figcaption>Add Inbound Rule 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*BvbF_f8Q5CGKoGCk.png" /><figcaption>Add Inbound Rule 2</figcaption></figure><p>Note: The network segment 0.0.0.0/0 is used only for exercises. You are not advised to use it in actual production because all requests from it are allowed to pass and therefore pose security risks.</p><p>After the security group is configured, return to the binding page and click <strong>OK</strong>. The Manager page is displayed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*IB-FdUtIGvvgN2aV.png" /><figcaption>Check the Network Error</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*AavdXB0WTLUvIxRU.png" /><figcaption>Bypass the Network Error</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*DBOdCEnIG9vcAKYM.png" /><figcaption>Admin Panel for MRS</figcaption></figure><p>Enter the username <strong>admin</strong> and its password <strong>Mrstest@123</strong> set during MRS cluster creation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*h_5aTVCVccs5kgha.png" /><figcaption>Admin Panel Monitoring</figcaption></figure><h3>Step 4: Create a Hive Table 🐝</h3><h4>What is Hive? 💡</h4><p><strong>Hive</strong> is a powerful data warehouse built on <strong>Hadoop</strong> that enables:</p><ul><li>⚙️ <strong>ETL (Extract, Transform, Load)</strong> operations</li><li>🔍 SQL-like queries on Hadoop data</li></ul><p>It is ideal for users familiar with SQL, providing a seamless way to query and analyze large datasets within the Hadoop ecosystem.</p><h4>What are Internal and External Tables? 📊</h4><p><strong>Internal Tables</strong> 🗂️:</p><ul><li>Data is <strong>fully managed</strong> by Hive.</li><li>Deleting an internal table also <strong>deletes the associated data</strong>.</li></ul><p><strong>External Tables</strong> 🌐:</p><ul><li>Data is <strong>managed externally</strong> and shared with other tools.</li><li>Deleting an external table removes only the <strong>metadata</strong>, keeping the data intact.</li></ul><p>With Hive’s table flexibility, you can choose the right type of table depending on how you plan to manage and share your data. 🚀</p><p><strong>1. Remote Login to ECS</strong></p><ul><li>Log in to the ECS console, locate the active ECS <strong>master1</strong>, and copy its EIP.</li><li>Use the terminal on the virtual desktop to log in to <strong>master1</strong> using ssh root@EIP with the password <strong>Mrstest@123</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*f1HaOoy1vXI17wVk.png" /><figcaption>ECS EIP Address</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*KbWOFiY83VA2Ukqd.png" /><figcaption>Open Terminal</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*-9_7EAfX89bmjDyV.png" /><figcaption>Connect to the EIP</figcaption></figure><p>(If a message is displayed, enter <strong>yes</strong> to proceed, and then enter the password to complete the login. Note that the entered password is not displayed.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/0*Y4G3T8DyM19Um9Fi.png" /><figcaption>Check the EIP Connection</figcaption></figure><p><strong>2. Create an Internal Table</strong></p><ul><li>Set environment variables using: source /opt/Bigdata/client/bigdata_env.</li><li>Enter <strong>beeline</strong> and press <strong>Enter</strong> to access Hive.</li><li>Run the following command to create an internal table:</li></ul><pre>CREATE TABLE default.test01 (<br> name STRING,<br> gender STRING,<br> age INT<br>)<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY &#39;,&#39;<br>STORED AS TEXTFILE;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/0*b1pE7mkd9ojS-r_y.png" /><figcaption>Source to bigdata env</figcaption></figure><p>Note that all statements in Hive must end with a semicolon (;). Otherwise, the statements cannot be executed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/0*xoq5LeDvVh2VoOlD.png" /><figcaption>Hive Access</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*mpQzzJ8xhzDhl_QJ.png" /><figcaption>Hive terminal</figcaption></figure><p>Run the <strong>show tables;</strong> command to display all existing tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*G6FpowljFc_pEapP.png" /><figcaption>Show table in Hive</figcaption></figure><p><strong>3. Create an External Table</strong></p><ul><li>Run the following command to create an external table:</li></ul><pre>CREATE EXTERNAL TABLE test02 (<br> name STRING,<br> gender STRING,<br> age INT<br>)<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY &#39;,&#39;<br>STORED AS TEXTFILE;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*1mJxBxalBkK56qtW.png" /><figcaption>Creating external table</figcaption></figure><p>Run the <strong>show tables;</strong> command to display all existing tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*vXe_ps_pGdaYMZeR.png" /><figcaption>Show table</figcaption></figure><p>With these steps, you should have successfully configured and managed your cluster on Huawei Cloud’s MRS platform.</p><h3>🗂️ HDFS, Hive, and Hadoop Analysis Guide</h3><p>This guide walks you through <strong>loading data into HDFS</strong>, <strong>querying data with Hive</strong>, and <strong>analyzing data using Hadoop</strong>. Follow these steps to master big data handling with ease! 🚀</p><h3>Step 5: Load HDFS Data 📥</h3><h4>What is HDFS? 💡</h4><p><strong>HDFS (Hadoop Distributed File System)</strong> is designed for reliable and distributed storage of massive amounts of data. Key features include:</p><ul><li>🛠️ <strong>Write Once, Read Multiple Times</strong>: Ideal for scenarios where data is written once and read repeatedly.</li><li>🔒 Ensures a file is written by only <strong>one caller at a time</strong> but can be read by <strong>multiple callers simultaneously</strong>.</li><li>⚡ Efficient for handling large-scale data storage and access.</li></ul><p>With HDFS, you gain robust, scalable, and distributed data storage tailored for big data processing. 🌟</p><p><strong>1. Edit the </strong><strong>test01.txt File<br></strong>Press <strong>Ctrl+C</strong> to exit Hive mode (or open a new terminal window) and edit the test01.txt file:</p><pre>vi test01.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/0*t3_VLANfwfJsxZd8.png" /><figcaption>Create txt file</figcaption></figure><p>Add the following data:</p><pre>hmeimei,female,20<br>jack,female,22<br>lilei,female,18<br>lily,male,23</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*lxeQju1w8oaQ8pB4.png" /><figcaption>Create new dataset</figcaption></figure><p>Press <strong>Esc</strong>, enter :wq, and press <strong>Enter</strong> to save and exit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*BAfJ0Q3gAkDhq0VH.png" /><figcaption>Save the dataset</figcaption></figure><p><strong>2. Verify File Contents<br></strong>Run the cat command to view the data in test01.txt:</p><pre>cat test01.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/0*4qCEsxmSa_rGbSna.png" /><figcaption>show the dataset</figcaption></figure><p><strong>3. Configure HDFS Environment Variables</strong></p><pre>source /opt/Bigdata/client/bigdata_env</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/0*SKyY2sIxtd9JwMfS.png" /><figcaption>Configure hdfs env</figcaption></figure><p><strong>4. Create a Directory in HDFS<br></strong>Create the test01 folder in the /user directory and verify it:</p><pre>hdfs dfs -mkdir /user/test01<br>hdfs dfs -ls /user/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/0*CO5V8HUcE3_s5ALu.png" /><figcaption>Check the folders</figcaption></figure><p><strong>5. Upload Data File to HDFS</strong></p><pre>hdfs dfs -put test01.txt /user/test01/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/0*hWK__4T1irhc_rAu.png" /><figcaption>Upload data files</figcaption></figure><p><strong>6. Import Data to the Hive External Table<br></strong>Enter Hive mode and run the following command:</p><pre>load data inpath &#39;/user/test01/test01.txt&#39; into table test02;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*6Ls8kN4iufjKzERr.png" /><figcaption>Load data</figcaption></figure><p><strong>7. View Data in the Hive Table</strong></p><pre>select * from test02;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*9dz7ctjzTidSjido.png" /><figcaption>View</figcaption></figure><h3>Step 6: Use Hive to Query Data⚡</h3><h3>1. Performing a Fuzzy Search</h3><p>Run the following command to search for tables with names starting with “test”:</p><pre>show tables like &#39;test*&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*y9pTD39RFqUqF7GN.png" /><figcaption>Show data</figcaption></figure><h3>2. Performing a Simple Search</h3><h4>Using the LIMIT Clause</h4><pre>select * from test02 limit 2;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*2oiERrYYtXiO5btr.png" /><figcaption>Show data using LIMIT</figcaption></figure><h4>Using the WHERE Clause</h4><pre>select * from test02 where gender =&#39;male&#39; limit 2;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*FxXqgVQ_msFKICOZ.png" /><figcaption>Using WHERE command</figcaption></figure><h4>Using the ORDER BY Clause</h4><pre>select * from test02 where gender =&#39;female&#39; order by age limit 2;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*sK8Gp9DYcVPG3fKM.png" /><figcaption>Using ORDER BY command</figcaption></figure><h3>3. Performing a Complex Query</h3><p>1. Edit the test03.txt file:</p><pre>vi test03.txt</pre><p>Add the following data:</p><pre>1001,Jack,Chinese,78<br>1002,Jack,English,79<br>1003,Jack,Math,80<br>1004,Mark,Chinese,79<br>1005,Mark,English,80<br>1006,Mark,Math,78<br>1007,Hanke,Chinese,80<br>1008,Hanke,English,78<br>1009,Hanke,Math,79</pre><p>Save and exit.</p><p>2. Upload the file to HDFS:</p><pre>hdfs dfs -put test03.txt /user/test01/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*tJ32ktJtR65ly9yj.png" /><figcaption>Put file to HDFS</figcaption></figure><p>3. Create a Hive table for the new data:</p><pre>create external table test03 (<br>   id int,<br>   name string,<br>   subject string,<br>   score float<br>) row format delimited fields terminated by &#39;,&#39; stored as textfile;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*fE4tm-5XiTCEsKax.png" /><figcaption>Create another table</figcaption></figure><p>4. Import data into the table:</p><pre>load data inpath &#39;/user/test01/test03.txt&#39; into table test03;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*31kOxKSxfsaXnyBC.png" /><figcaption>Load data</figcaption></figure><p>5. Query data from the table:</p><pre>select * from test03;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*Z2w-kH75QfIog1VZ.png" /><figcaption>Select from data</figcaption></figure><h4>Aggregation Queries</h4><p>1. Summarize scores:</p><pre>select name, sum(score) as total_score from test03 group by name;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*UF0vn8vMlAkLab9P.png" /><figcaption>Summarize the scores</figcaption></figure><p>2. Filter results:</p><pre>select name, sum(score) as total_score from test03 group by name having total_score &gt; 230;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*urw0o-K9bMtDxuUf.png" /><figcaption>Filter the results</figcaption></figure><p>3. Find the highest score per subject:</p><pre>select subject, max(score) from test03 group by subject;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*-orER-ZHMhwkzdid.png" /><figcaption>Select the highest score</figcaption></figure><p>4. Count participants per subject:</p><pre>select subject, count(1) from test03 group by subject;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*HQ91vc4xcREwKcZw.png" /><figcaption>Count participants per subject</figcaption></figure><h3>Step 7: Use Hadoop to Analyze Data 📊</h3><h4>What is Hadoop? 💡</h4><p><strong>Hadoop</strong> is an open-source framework designed for distributed storage and processing of large data sets. With its ecosystem of tools, it enables:</p><ul><li>⚙️ <strong>Parallel Data Processing</strong> across multiple nodes.</li><li>🔍 Scalability for analyzing massive datasets efficiently.</li><li>🛠️ Support for a wide range of data analysis operations using tools like MapReduce.</li></ul><h3>1. Preparing a Hadoop Sample Program</h3><p>To prepare the sample program, follow these steps:</p><p>1. Exit Hive mode by pressing <strong>Ctrl+C</strong> and download the Hadoop sample program using the following command:</p><pre>wget https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*ZmSeMS2ZT_ViMd_-.png" /><figcaption>Get Hadoop</figcaption></figure><p>2. Decompress the downloaded package:</p><pre>tar -zxvf hadoop-3.2.4.tar.gz</pre><h3>2. Preparing Hadoop Data Files</h3><ol><li>Edit the <strong>wordcount1.txt</strong> and <strong>wordcount2.txt</strong> files:</li></ol><pre>vi wordcount1.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/0*CarmquZFxRkigH4r.png" /><figcaption>Preparing new data 1</figcaption></figure><pre>vi wordcount2.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/0*nm0_es5HgmVeIU0Z.png" /><figcaption>Preparing new data 2</figcaption></figure><p>2. Create the <strong>helloworld</strong> folder in the HDFS root directory and verify it:</p><pre>hdfs dfs -mkdir /user/helloworld<br>hdfs dfs -ls /user/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/0*CRxASRrSZ2rcsiGy.png" /><figcaption>Check the folder</figcaption></figure><p>3. Create the <strong>input</strong> folder in the HDFS root directory:</p><pre>hdfs dfs -mkdir /user/input<br>hdfs dfs -ls /user/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/0*AvzAlYWkKKJO96qW.png" /><figcaption>Check the folder</figcaption></figure><p>4. Upload the sample program to the <strong>helloworld</strong> folder:</p><pre>hdfs dfs -put hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.4.jar /user/helloworld/</pre><p>5. Upload the edited <strong>wordcount1.txt</strong> and <strong>wordcount2.txt</strong> files to the <strong>input</strong> folder:</p><pre>hdfs dfs -put wordcount1.txt /user/input/<br>hdfs dfs -cat /user/input/wordcount1.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*avBYqaPooSP81Cqm.png" /><figcaption>Upload files to input</figcaption></figure><pre>hdfs dfs -put wordcount2.txt /user/input/<br>hdfs dfs -cat /user/input/wordcount2.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/0*lkFYbCez5uSXvD_D.png" /><figcaption>Upload files to input</figcaption></figure><h3>3. Submitting an Analysis Job on the MRS Console</h3><ol><li>In the MRS console, go to <strong>Clusters &gt; Active Clusters</strong>. Click the <strong>mrs_test</strong> cluster.</li><li>Click <strong>Jobs</strong>, then <strong>Create</strong> to open the <strong>Create Job</strong> dialog box.</li></ol><ul><li><strong>Type</strong>: MapReduce</li><li><strong>Name</strong>: wordcount</li><li><strong>Program Path</strong>: Click <strong>HDFS</strong> and select the uploaded sample program. The path is:</li></ul><pre>/user/helloworld/hadoop-mapreduce-examples-3.2.4.jar</pre><ul><li><strong>Parameters</strong>:</li></ul><pre>wordcount hdfs://hacluster/user/input/ hdfs://hacluster/user/output/</pre><ul><li><strong>Note</strong>: Ensure the output directory does not already exist.</li><li><strong>Service Parameter</strong>: Leave it blank.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*L4RPf7LUCukZWKj6.png" /><figcaption>Check active Cluster</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*UT8tWrV6PJNsDkvL.png" /><figcaption>Create new jobs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*sA4_FZ0ghyeadAe7.png" /><figcaption>Configure job settings</figcaption></figure><p>3. Click <strong>OK</strong> to submit the job.</p><p>4. In the <strong>Jobs</strong> tab, monitor the job execution status and logs. After several minutes, the job will complete successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/0*yrESsbD08s8izvoV.png" /><figcaption>Check the status of job</figcaption></figure><p>5. Verify the output:</p><ul><li>Run the following command to list the files in the output directory:</li></ul><pre>hdfs dfs -ls /user/output</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*xQsRYCKl44vXEisD.png" /><figcaption>Check the output</figcaption></figure><ul><li>Run the following command to view the results in the part-r-00000 file:</li></ul><pre>hdfs dfs -cat /user/output/part-r-00000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*0yZxqWcifGgSvB0l.png" /><figcaption>Check the result of job</figcaption></figure><p>🙏 <strong>Thank You for Reading!</strong></p><p>We hope this guide has been helpful in navigating <strong>HDFS, Hive, and Hadoop analysis</strong> on Huawei Cloud’s MRS platform. Your journey in mastering big data is just beginning, and we’re excited to see what you’ll achieve! 🚀</p><p>Feel free to revisit this guide anytime, and don’t hesitate to explore more about these powerful tools. 🌟</p><p>Happy learning and analyzing! 💡📊</p><p><strong>References:</strong></p><ul><li><a href="https://www.huaweicloud.com/intl/en-us/product/mrs.html">MapReduce Service (MRS) - Big Data Components | Huawei Cloud</a></li><li><a href="https://hive.apache.org/docs/latest/">Space Details:</a></li><li><a href="https://www.geeksforgeeks.org/apache-hive/">Apache Hive - GeeksforGeeks</a></li><li><a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">HDFS Users Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb9bce72a2cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/huawei-developers/%EF%B8%8F-build-manage-and-analyze-the-ultimate-guide-to-huawei-cloud-mrs-platform-bb9bce72a2cd">☁️ Build, Manage, and Analyze: The Ultimate Guide to Huawei Cloud MRS Platform</a> was originally published in <a href="https://medium.com/huawei-developers">Huawei Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create your own customizable Chatbot on Huawei Cloud ☁️]]></title>
            <link>https://medium.com/huawei-developers/create-your-own-customizable-chatbot-on-huawei-cloud-%EF%B8%8F-d44e5291ab02?source=rss-feb0d1d6abf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d44e5291ab02</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[huawei]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Berhan Ay]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 11:42:39 GMT</pubDate>
            <atom:updated>2023-11-06T11:42:39.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xy-9Y0lP7NCp3AnFHPglfQ.png" /><figcaption>Cover of Article</figcaption></figure><h3>The Simple Way to Create a Chatbot🤖 with Huawei Cloud</h3><p>Hello everyone! 👋😊Today, I’m going to show you how to create your very own chatbot in just a few simple steps.🤗☕🍫</p><p>Nowadays, chatbots have become an indispensable part of improving customer service for businesses and developers. These chatbots are able to communicate with users in a smart and effective way by utilizing the power of large language models. Cloud services like Huawei Cloud provide the infrastructure you need to create your own chatbot.🧠</p><p>Large language models are highly capable artificial intelligence systems in generating and understanding text. In this article, you will learn how to create a chatbot step by step using Huawei Cloud and how you can utilize large language models. Additionally, you will discover how your chatbot can contribute to your business and improve the user experience.</p><p>Creating your own chatbot is not only a great way to improve customer services but also a fantastic way to empower your business with automation and increase efficiency. With the robust infrastructure provided by Huawei Cloud, reaching these goals becomes more accessible.</p><p>Let’s get started and move on to the steps to create your own chatbot on Huawei Cloud!</p><h3>Step 1: Creating a Huawei Cloud Account</h3><ul><li>Go to the official Huawei Cloud website.</li><li>Click on the “Sign Up” option to open a new account.</li><li>Fill in the requested information and verify your account via email or phone number.</li></ul><h3>Step 2: Setting Up a GPU-Supported Virtual Machine</h3><ul><li>Log in to the Huawei Cloud console.</li><li>Go to the “Compute” and “Elastic Cloud Server” menu.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3OIECr3saSM41cwe5rKTw.png" /><figcaption>ECS</figcaption></figure><ul><li>Find and click the “Create ECS” or “Launch New ECS” option.</li><li>Fill in the necessary details completely:</li><li><strong>Region and Availability Zone:</strong> Select your preferred region.</li><li><strong>Server Type:</strong> Choose a server type with GPU accelerated.</li><li><strong>Operating System:</strong> Preferably choose Ubuntu 22.04.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1f332Z0CQIMV1rEUalW8rg.png" /><figcaption>Creating GPU Accelerated ECS</figcaption></figure><ul><li><strong>Network Settings:</strong> Configure options for VPC, Subnet, Elastic IP, and Security Group.</li><li>After completing the other required settings, press the “Create” or “Start” button to set up your server.</li></ul><h3>Step 3: Providing Internet Access with an Elastic IP</h3><ul><li>Navigate to the “Elastic IP” section.</li><li>Click on “Allocate Elastic IP” or “Create New Elastic IP.”</li><li>Assign the Elastic IP you created to your new virtual machine to provide internet access.</li></ul><h3>Step 4: Security Group Settings</h3><ul><li>Go to the “Security Group” section.</li><li>Create a new Security Group or edit an existing one.</li><li>Set the incoming connection rules as follows:</li><li><strong>Protocol:</strong> Allow SSH (port 22) and HTTP (port 80).</li><li><strong>Source IP:</strong> Specify IP addresses that you are sure of their security for SSH.</li><li>Assign your created Security Group to your server.</li></ul><h3>Step 5: Connecting to the Server</h3><ul><li>Use an SSH client to connect to your virtual machine.</li><li>Make an SSH connection using the Elastic IP address obtained in previous steps.</li><li>You can use any program like mobaXterm or putty or you can ssh on your local terminal as ssh root@(your elastic ip)</li><li>Enter your server username and password.</li></ul><h3>Step 6: Installing Conda and Creating an Environment</h3><pre># Download and install Miniconda<br>curl -sL &quot;https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh&quot; &gt; &quot;Miniconda3.sh&quot;<br>bash Miniconda3.sh</pre><pre># Create a new conda environment<br>conda create -n chatbot python=3.11<br>conda activate chatbot</pre><h3>Step 7: Installing PyTorch and Dependencies</h3><p>The installation of PyTorch can vary depending on whether you have a GPU, and if so, what type. The following commands illustrate PyTorch installation for different systems:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8822d58661f3917e37787fa53b6ec31/href">https://medium.com/media/f8822d58661f3917e37787fa53b6ec31/href</a></iframe><p>Once Miniconda is installed, create a new conda environment:</p><pre>conda install -y -c nvidia/label/cuda-11.3.0 cuda-runtime</pre><h3>Step 8: Setting Up a Web Interface</h3><p>If you wish to create a web interface for your chatbot, you can clone this repository (it is an opensource large language model UI) and install the necessary dependencies using the following commands:</p><pre>git clone https://github.com/oobabooga/text-generation-webui</pre><pre>cd text-generation-webui</pre><p>Use the appropriate requirements file based on your system:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9cbe2898b7c6ff89cd4f2646b5697812/href">https://medium.com/media/9cbe2898b7c6ff89cd4f2646b5697812/href</a></iframe><pre>pip install -r requirements.txt</pre><p>To start the chatbot UI:</p><pre>conda activate chatbot<br>cd text-generation-webui<br>python server.py --share</pre><p><strong>Login to the Chatbot Site</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RNWNDwhoXdKSZssh60ByTQ.png" /><figcaption>LLM Loading to Web App</figcaption></figure><p><strong>Step 1: </strong>Choose a pre-trained model from Hugging Face, like llama, open-chat, mistral etc.</p><p><strong>Step 2: </strong>Click ‘Get file list’ to view available versions of the selected model.</p><p><strong>Step 3: </strong>Write the exact name of the model you intend to use.</p><p><strong>Step 4: </strong>Click the download button for the chosen model to download it.</p><p><strong>Step 5: </strong>Locate and select the downloaded model in your local directory.</p><p><strong>Step 6: </strong>Load the model for your application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mNRPUFzwwdL_jA6iaM7E3Q.png" /><figcaption>Configuration of ChatBot</figcaption></figure><p><strong>Character:</strong> This tab is currently selected and appears to let you configure the persona of the chatbot. In this case, the persona is Albert Einstein, and the user can interact with the chatbot as if they were speaking to Einstein himself.</p><p><strong>Character picture:</strong> The image representing Albert Einstein, enhances the immersion of the conversation.</p><p><strong>Your picture:</strong> An image of the user, which might be used to personalize the chat session.</p><p><strong>Character/Your name:</strong> Fields where the names are specified for the character (Albert Einstein) and the user.</p><p><strong>Context:</strong> An area where the context or backstory for the character is provided, which the chatbot presumably uses to tailor its responses to match the persona of Albert Einstein.</p><p><strong>Albert Einstein Bot </strong>‍🔬<strong>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pPr-jonnsk2f5I6dSBKfDQ.png" /><figcaption>Chat with Albert Einstein</figcaption></figure><p>This particular interface allows for a simulated chat with a character designed to represent Albert Einstein. The chat is interactive, with the user typing in questions or comments and the Einstein chatbot responding.</p><p><strong>Huawei Cloud DTSE Bot:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4jtkNEA6H8hpg5hxSk5WQ.png" /><figcaption>Chat with DTSE Bot</figcaption></figure><p>This chatbot is presented as a tech enthusiast with a calm demeanor and an ability to simplify complex technical concepts.</p><p>In the conversation, the user asks Kenji to suggest a web application architecture for using Huawei Cloud Services. Kenji DTSE responds with a list of five recommendations tailored to the Huawei Cloud environment.</p><h3>Conclusion</h3><p>The project’s journey has led us to successfully establish a sophisticated chatbot on Huawei Cloud, leveraging the power of GPU-accelerated virtual machines and the robustness of Python’s ecosystem. By utilizing Conda environments and installing PyTorch, we’ve set the stage for advanced machine learning capabilities. The chatbot, poised to enhance customer interaction and service, stands on the cutting edge of conversational AI. The web interface integration promises a seamless user experience, making the chatbot accessible and user-friendly. As businesses continue to evolve, the chatbot created in this project represents a significant step towards embracing AI-driven communication, ensuring that businesses stay ahead in the dynamic digital landscape.</p><p>References</p><ul><li><a href="https://www.huaweicloud.com/intl/en-us/">Huawei Cloud - The AI Pioneer in Industries | Huawei Cloud</a></li><li><a href="https://github.com/oobabooga/text-generation-webui">GitHub - oobabooga/text-generation-webui: The definitive Web UI for local AI, with powerful features and easy setup.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d44e5291ab02" width="1" height="1" alt=""><hr><p><a href="https://medium.com/huawei-developers/create-your-own-customizable-chatbot-on-huawei-cloud-%EF%B8%8F-d44e5291ab02">Create your own customizable Chatbot on Huawei Cloud ☁️</a> was originally published in <a href="https://medium.com/huawei-developers">Huawei Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to NLP Service on Huawei Cloud]]></title>
            <link>https://medium.com/huawei-developers/introduction-to-nlp-service-on-huawei-cloud-2c9b9764230e?source=rss-feb0d1d6abf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c9b9764230e</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[huawei-cloud]]></category>
            <category><![CDATA[huawei]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Berhan Ay]]></dc:creator>
            <pubDate>Tue, 04 Apr 2023 12:40:55 GMT</pubDate>
            <atom:updated>2023-04-04T12:43:22.043Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HfyYo9j5fcdUk07r54guaw.jpeg" /><figcaption>NLP</figcaption></figure><h3>Introduction</h3><p>Hi 👋😉, in this article I will describe you Natural Language Processing of Huawei Cloud services. Enjoyable reading <strong>❗ 🤗☕🍫</strong></p><p><strong><em>NLP</em></strong> is an area of artificial intelligence that focuses on natural language interaction between computers and people. It entails a variety of approaches and algorithms that allow machines to comprehend, interpret, and synthesize human language.</p><p>The service provides a variety of features that may be used in a variety of fields, including intelligent <strong>Q&amp;A</strong>, <strong>content analysis</strong>, <strong>sentiment analysis</strong>, and <strong>content suggestion</strong>.​</p><p>NLP processes large volumes of unstructured text data using machine learning and deep learning algorithms, making it <strong>simpler</strong> for users to extract valuable insights and information from them.</p><p>It is also well-known for its capacity to automate processes that would normally need human intervention, such as language translation and text summary. NLP is an essential tool for businesses that deal with large amounts of textual data and need to extract insights quickly and efficiently.</p><p><strong>Natural Language Processing (NLP)</strong> Foundations provide APIs for natural language processing such as word segmentation, named entity recognition (NER), keyword extraction, short text comparison, and more. It may be used in a variety of scenarios, including intelligent Q&amp;A, public opinion analysis, and content suggestion.</p><h3>Sub-Services:</h3><p><strong>NLP consists of the following sub-services:</strong></p><ul><li><strong>Language Generation (LG)</strong> provides language generation APIs that may be used to generate texts such as news summaries, document summaries, search result segments, and product review summaries.</li><li><strong>Language Understanding (LU)</strong> offers language understanding APIs such as text categorization and sentiment analysis. It may be used to analyze emotions, detect content, and recognize advertisements.</li><li><strong>Machine Translation (MT)</strong> eliminates communication barriers by providing consumers with speedy and accurate translation services. It may be used to translate documents.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*VaYgVmY1Y42RQDkt.png" /><figcaption>Language Generation Progress</figcaption></figure><h3>Advantages</h3><p>The breadth of NLP’s features provides customers with traditional natural language algorithms and solutions that may answer a wide variety of needs in many fields.</p><p>Developers can, for example, employ NLP to construct intelligent Q&amp;A systems, content analysis, and content recommendation systems.</p><p>In terms of <strong>efficiency and accuracy</strong>, NLP can swiftly and <strong>reliably</strong> extract crucial information from <strong>large-sized texts</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/0*V31_RJT9V5ftPe2S.png" /><figcaption>Advantages of NLP Service</figcaption></figure><p>​</p><p>This capacity is especially beneficial for jobs like content summary, in which NLP may produce summaries of articles, news, search results, and product evaluations without altering their original meaning.</p><p>Another key advantage of NLP is its <strong>ease of usage</strong>, since it offers consumers with simple APIs. This eliminates the requirement for developers to download SDK files or acquire servers. Cross-platform calling is also available, and the service offers a wide selection of goods with single-point provisioning and deployment.</p><p>Lastly, NLP is consistent and dependable. It offers fault-triggered migration, ensuring <strong>99.95% service availability</strong> and service continuity. Numerous data copies further assure data durability of <strong>99.99995%, preventing data loss</strong>.</p><h3>Fundamental Concepts</h3><p>Word segmentation, NER, text similarity, text summarization, and sentiment analysis are examples of NLP principles.</p><p><strong>Word segmentation</strong> divides the text into separate words and tags the acquired words with part-of-speech tags.</p><p><strong>NER</strong> detects text entities such as human names, locations, organizations, proper nouns, and more.</p><p><strong>Text similarity</strong> computes text similarity.</p><p><strong>Text summarization</strong> summarizes a document’s essential material without affecting its original meaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/0*iiLQ2uhzROBbYlM7.png" /><figcaption>Applications of NLP</figcaption></figure><p>​</p><h3>Obtaining NLP</h3><p>NLP is accessible via a web-based administration dashboard or HTTPS-based APIs. Users can select the suitable mode based on their requirements.</p><p>Using the management console: You can enable NLP on the <strong>management console</strong>.</p><p>Using APIs: NLP provides users with open APIs, which can be invoked for users to <strong>integrate</strong> NLP into <strong>third-party systems</strong>.</p><ul><li><strong>cURL</strong></li></ul><p>cURL is a command-line tool used to perform URL operations and transfer files. cURL acts as the HTTP client that can send HTTP requests to the HTTP server and receive response messages. It is applicable to API debugging.</p><ul><li><strong>Code</strong></li></ul><p>invoke APIs using code to assemble, send, and process request messages.</p><ul><li><strong>REST client</strong></li></ul><p>Both Mozilla Firefox and Google Chrome provide a graphical browser plugin, that is, REST client, to send and process requests.</p><h3>Conclusion</h3><blockquote><em>NLP is a strong cloud-based solution that provides customers with text analysis and mining capabilities. It can assist improve the efficiency and accuracy of text processing in a variety of disciplines, including intelligent Q&amp;A, content analysis, and content recommendation. NLP is an useful tool to organizations and developers because of its simple APIs and robust and trustworthy service.</em></blockquote><h3>References 📜📚</h3><p><strong>1.Huawei Cloud NLP Service</strong></p><p><a href="https://support.huaweicloud.com/intl/en-us/nlp/index.html">Progressive Knowledge_Natural Language Processing_Huawei Cloud</a></p><p><strong>2.Intelligent Agents</strong></p><p><a href="https://www.huawei.com/en/huaweitech/publication/winwin/ai/intelligent-agents-tomorrow-digital-valets">Natural language processing: Intelligent agents - Huawei Publications</a></p><p><strong>3. Natural Language Processing Guide</strong></p><p><a href="https://www.deeplearning.ai/resources/natural-language-processing/">Natural Language Processing (NLP) - A Complete Guide</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c9b9764230e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/huawei-developers/introduction-to-nlp-service-on-huawei-cloud-2c9b9764230e">Introduction to NLP Service on Huawei Cloud</a> was originally published in <a href="https://medium.com/huawei-developers">Huawei Developers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>