<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Data Science Pro on Medium]]></title>
        <description><![CDATA[Stories by Data Science Pro on Medium]]></description>
        <link>https://medium.com/@yoshiyoshiyoshi?source=rss-1c9ce7556478------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Data Science Pro on Medium</title>
            <link>https://medium.com/@yoshiyoshiyoshi?source=rss-1c9ce7556478------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:37:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yoshiyoshiyoshi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[FRED , Time Series Lab , USDJPY & Flitring with Grok]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/fred-time-series-lab-usdjpy-flitring-with-grok-707018c20241?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/707018c20241</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 19:27:07 GMT</pubDate>
            <atom:updated>2025-04-07T21:06:39.232Z</atom:updated>
            <content:encoded><![CDATA[<h3>FRED , Time Series Lab , USDJPY , Derivatives &amp; Flitring with Grok</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NQSTqWy-S5czOwy_" /></figure><p><a href="https://www.timeserieslab.com/">Time Series Lab — Advanced Time Series Forecasting Software</a></p><h4>FRED — St.Louis</h4><p>To explore the wealth of data available in the Federal Reserve Economic Data (FRED) database and understand what information you can access via your API keys in Time Series Lab, you can start by browsing the FRED website directly. The FRED database, maintained by the Federal Reserve Bank of St. Louis, offers a vast collection of economic time series data, and its structure can guide you in discovering what’s available for your analysis. Here’s how you can see what’s available on FRED:</p><p><strong>1. Visit the FRED Website</strong><br>Go to [fred.stlouisfed.org](<a href="https://fred.stlouisfed.org/">https://fred.stlouisfed.org/</a>). This is the primary interface for exploring the database, and it’s freely accessible even without an API key for browsing purposes.</p><p><strong>2. Browse by Categories</strong><br>On the FRED homepage, you’ll find a section labeled **”Categories”** or a similar option to browse data. Click on it to see a broad list of economic data topics, such as:<br>- **Money, Banking, &amp; Finance**: Interest rates, monetary aggregates, banking statistics.<br>- **National Accounts**: GDP, personal income, government spending.<br>- **Population, Employment, &amp; Labor Markets**: Unemployment rates, payroll data, labor force participation.<br>- **Prices**: Consumer Price Index (CPI), Producer Price Index (PPI), inflation metrics.<br>- **Production &amp; Business Activity**: Industrial production, retail sales, housing starts.<br>- **International Data**: Exchange rates, foreign GDP, trade balances.<br>- **U.S. Regional Data**: State-level employment, housing prices, etc.</p><p>Each category expands into subcategories, giving you a hierarchical view of the data available. For example, under “Prices,” you might find specific series like “CPI for All Urban Consumers” or “PPI for Final Demand.”</p><p><strong>3. Search by Keywords</strong><br>Use the search bar at the top of the FRED homepage. Type in keywords related to the data you’re interested in (e.g., “unemployment,” “inflation,” “GDP,” “interest rates”). This will return a list of relevant time series, each with a unique **Series ID** (e.g., “UNRATE” for the U.S. unemployment rate). These Series IDs are what you’ll use in Time Series Lab with the API to fetch specific datasets.</p><p><strong>4. Explore by Source or Release</strong><br>FRED organizes data by **Source** (e.g., Bureau of Labor Statistics, U.S. Census Bureau) and **Release** (e.g., Employment Situation, Consumer Price Index). From the homepage, click on **”Sources”** or **”Releases”** to see the organizations and reports contributing data. This can help you identify authoritative datasets for your analysis.</p><p><strong>5. Check Popular Series</strong><br>On the homepage, FRED highlights **”Popular Series”** like the Federal Funds Rate, Real GDP, or CPI. These give you a quick sense of commonly used datasets that might be a good starting point.</p><p><strong>6. Use the FRED API Documentation</strong><br>Since you’re using the API with Time Series Lab, visit the [FRED API documentation](<a href="https://fred.stlouisfed.org/docs/api/fred/">https://fred.stlouisfed.org/docs/api/fred/</a>). It provides endpoints like:<br>- **`fred/category`**: Retrieve a list of categories and subcategories.<br>- **`fred/series/search`**: Search for series by keywords or tags.<br>- **`fred/releases`**: Get a list of data releases.</p><p>You can test these endpoints directly on the FRED website or integrate them into Time Series Lab to programmatically explore available data. For example, querying the `fred/category` endpoint will return a structured list of all categories, mirroring what you see on the website.</p><p><strong>7. Leverage FRED’s Tools</strong><br>- **GeoFRED**: If you’re interested in regional data, check [GeoFRED](<a href="https://geofred.stlouisfed.org/">https://geofred.stlouisfed.org/</a>) for maps of U.S. state and county-level data.<br>- **ALFRED**: For vintage data (historical revisions), explore [ALFRED](<a href="https://alfred.stlouisfed.org/">https://alfred.stlouisfed.org/</a>).</p><p><strong>8. Practical Steps in Time Series Lab</strong><br>Once you’ve identified Series IDs or categories on the FRED website, use them in Time Series Lab. For example:<br>- Search for “UNRATE” (unemployment rate) or “GDP” to pull those series.<br>- If the software supports category browsing, use the API to fetch a list of series within a category (e.g., all labor market data).</p><p>What’s Available in General?<br>FRED contains over 825,000 time series as of early 2025, covering:<br>- **U.S. and International Macroeconomic Data**: GDP, inflation, employment, trade.<br>- **Financial Data**: Interest rates, stock market indices, exchange rates.<br>- **Regional Data**: State and county-level economic indicators.<br>- **Frequencies**: Daily, weekly, monthly, quarterly, annual.<br>- **Sources**: Government agencies (e.g., BLS, BEA), international organizations (e.g., OECD, IMF), and private entities.</p><p>To get a comprehensive overview, download the full list of series from the FRED website (under “Data Tools” &gt; “Download Data”) as a CSV file, or use the API’s `fred/series` endpoint to retrieve metadata programmatically.</p><p>By starting on the FRED website and experimenting with searches or category browsing, you’ll quickly get a sense of the database’s scope and pinpoint the data relevant to your needs in Time Series Lab.</p><p>The Federal Reserve Economic Data (FRED) database does not provide real-time or historical data for <strong>forex pairs </strong>in the traditional sense, such as bid/ask prices or intraday trading data (e.g., EUR/USD, GBP/JPY) that you’d typically find on forex trading platforms like MetaTrader or Bloomberg.</p><p>FRED focuses primarily on economic time series data rather than market-specific trading data. However, it does offer a variety of **exchange rate data**, which can include daily, monthly, or annual averages for currency pairs, typically from official sources like central banks or government agencies.</p><h4>Here’s what FRED offers related to forex pairs and exchange rates:</h4><p><strong>1. Exchange Rate Data on FRED</strong><br>FRED provides exchange rate series for many currencies relative to the U.S. dollar (USD), as well as some cross rates. These are not “forex pairs” in the high-frequency, trader-oriented sense but rather economic indicators often reported by institutions like the Federal Reserve, the International Monetary Fund (IMF), or other statistical agencies. Examples include:<br>- **Nominal Exchange Rates**: Daily or monthly averages of currencies against the USD (e.g., USD/EUR, USD/JPY, USD/GBP).<br>- **Real Exchange Rates**: Adjusted for inflation differentials between countries.<br>- **Trade-Weighted Indexes**: Broader measures of a currency’s value against a basket of trading partners’ currencies (e.g., U.S. Dollar Index).</p><p><strong>2. Examples of Available Series</strong><br>To give you a sense of what’s available, here are some specific exchange rate series you can find on FRED (search by Series ID or description):<br>- **DEXUSEU**: U.S. Dollars per Euro (daily, noon buying rates in New York, from the Federal Reserve).<br>- **DEXJPUS**: Japanese Yen per U.S. Dollar (daily).<br>- **DEXUSUK**: U.S. Dollars per British Pound (daily).<br>- **TWEXB**: Trade Weighted U.S. Dollar Index (Broad Index, weekly or monthly).<br>- **REER**: Real Effective Exchange Rate for the United States (from IMF or BIS data).</p><p>These series are typically daily or monthly averages, not tick-by-tick forex data, and are sourced from the Federal Reserve Board, IMF, or other official entities.</p><p><strong>3. How to Check for Forex-Related Data</strong><br>- **On the FRED Website**: Go to [fred.stlouisfed.org](<a href="https://fred.stlouisfed.org/">https://fred.stlouisfed.org/</a>), and in the search bar, type keywords like “exchange rate,” “USD/EUR,” “yen,” or “trade-weighted dollar.” You can also browse the **”International Data”** category, then select **”Exchange Rates”**.<br>- **Via API**: Use the `fred/series/search` endpoint with your API key in Time Series Lab. For example, search for “exchange rate” or specific pairs like “USD/EUR” to retrieve matching Series IDs.<br>- **Sources**: Look under releases like “H.10 Foreign Exchange Rates” (Federal Reserve) or “International Financial Statistics” (IMF) for available data.</p><p><strong>4. Limitations for Forex Traders</strong><br>If you’re looking for forex pairs with:<br>- **High-frequency data** (e.g., minute-by-minute or tick data),<br>- **Bid/ask spreads**, or<br>- **Cross-currency pairs not involving USD** (e.g., EUR/JPY directly),<br>FRED won’t meet those needs. For such data, you’d need a dedicated forex data provider like OANDA, Dukascopy, or a broker’s historical data feed.</p><p><strong>5. What You Can Do With FRED Data</strong><br>While not tailored for active forex trading, FRED’s exchange rate data is excellent for:<br>- Economic analysis (e.g., studying currency trends alongside GDP or inflation).<br>- Time series modeling in tools like Time Series Lab.<br>- Historical comparisons of currency strength (e.g., using real effective exchange rates).</p><p>As a USD/JPY trader, leveraging FRED data can enhance your understanding of the macroeconomic factors driving this forex pair. While FRED doesn’t provide real-time tick data or bid/ask spreads, it offers key economic time series that influence the U.S. dollar (USD) and Japanese yen (JPY) exchange rate over time.</p><blockquote><strong><em>USD/JPY is heavily impacted by interest rate differentials, inflation expectations, and monetary policy signals from the Federal Reserve (Fed) and the Bank of Japan (BoJ).</em></strong></blockquote><h4><strong>Key FRED Metrics for USD/JPY Trading</strong></h4><p><strong>1. Exchange Rate Baseline</strong><br>- **Series ID: DEXJPUS**<br> — **Description**: Japanese Yen per U.S. Dollar (daily, noon buying rates in New York).<br> — **Why It Matters**: This is the USD/JPY exchange rate itself, giving you a historical benchmark to correlate with other economic indicators. A rising DEXJPUS means JPY is weakening (more yen per dollar), often tied to U.S. strength or Japanese policy easing.</p><p><strong>2. Interest Rates (U.S. and Japan)</strong><br>Interest rate differentials are a primary driver of USD/JPY. When U.S. rates rise relative to Japanese rates, USD tends to strengthen (USD/JPY rises), and vice versa.</p><p>- **U.S. Interest Rates**<br> — **Series ID: FEDFUNDS**<br> — **Description**: Effective Federal Funds Rate (monthly average).<br> — **Why It Matters**: This is the Fed’s key policy rate. Higher rates attract capital to USD-denominated assets, strengthening the dollar against the yen. Watch for Fed tightening cycles (rate hikes) as a bullish signal for USD/JPY.<br> — **Series ID: DGS10**<br> — **Description**: 10-Year Treasury Constant Maturity Rate (daily).<br> — **Why It Matters**: Long-term U.S. yields reflect market expectations of growth and inflation. A widening gap between U.S. 10-year yields and Japanese yields (see below) often boosts USD/JPY. Traders use this as a proxy for carry trade profitability.<br> — **Series ID: DGS2**<br> — **Description**: 2-Year Treasury Constant Maturity Rate (daily).<br> — **Why It Matters**: Short-term yields are more sensitive to near-term Fed policy expectations. Compare this to Japan’s short-term rates for quick shifts in differentials.</p><p>- **Japanese Interest Rates**<br> — **Series ID: IRSTCI01JPM156N** (or similar, check availability)<br> — **Description**: Japan Government Bond Yield, 10-Year (monthly, from OECD or IMF data).<br> — **Why It Matters**: Japan’s ultra-low or negative yield environment (due to BoJ’s Yield Curve Control) contrasts with U.S. yields. A stable or declining Japanese 10-year yield versus rising U.S. yields supports USD/JPY appreciation.<br> — **Note**: Japan’s short-term rates (e.g., overnight call rate) are often near zero or negative and less dynamic on FRED. The BoJ’s policy rate isn’t as directly accessible, but the 10-year yield captures their influence.</p><p>- **Key Analysis**: Calculate the **U.S.-Japan yield differential** (e.g., DGS10 minus Japan’s 10-year yield). A widening differential favors USD/JPY upside, especially in carry trade strategies.</p><p><strong>3. Inflation (U.S. and Japan)</strong><br>Inflation affects real interest rates and central bank policy, indirectly driving USD/JPY through expectations of tightening or easing.</p><p>- **U.S. Inflation**<br> — **Series ID: CPIAUCSL**<br> — **Description**: Consumer Price Index for All Urban Consumers (CPI-U), Seasonally Adjusted (monthly).<br> — **Why It Matters**: Headline CPI reflects broad price pressures in the U.S. Rising CPI may prompt Fed rate hikes, strengthening USD against JPY. Focus on year-over-year changes.<br> — **Series ID: CPILFESL**<br> — **Description**: Core CPI (excludes food and energy, monthly).<br> — **Why It Matters**: Core inflation is a key metric for the Fed, smoothing out volatile components. Persistent core CPI increases signal sustained Fed tightening, bullish for USD/JPY.<br> — **Series ID: PCEPI**<br> — **Description**: Personal Consumption Expenditures Price Index (monthly).<br> — **Why It Matters**: The Fed’s preferred inflation gauge. If PCE rises above the 2% target, expect hawkish policy, supporting USD strength.</p><p>- **Japanese Inflation**<br> — **Series ID: JPNCPIALLMINMEI**<br> — **Description**: Consumer Price Index, All Items, Japan (monthly, from IMF/OECD).<br> — **Why It Matters**: Japan struggles with low inflation or deflation. Weak CPI reinforces BoJ’s dovish stance (low rates), weakening JPY relative to USD. Compare this to U.S. CPI for relative pressure.<br> — **Series ID: JPNCPICORMINMEI**<br> — **Description**: Core CPI for Japan (excludes food/energy, monthly).<br> — **Why It Matters**: Like the U.S., core CPI shows underlying trends. Persistently low Japanese core inflation keeps BoJ policy loose, favoring USD/JPY upside.</p><p>- **Key Analysis**: Monitor the **U.S.-Japan inflation differential** (e.g., CPIAUCSL minus JPNCPIALLMINMEI). A growing gap suggests U.S. tightening outpacing Japan’s stagnation, lifting USD/JPY.</p><p><strong>4. Other Relevant Metrics</strong><br>- **Series ID: UNRATE**<br> — **Description**: U.S. Unemployment Rate (monthly).<br> — **Why It Matters**: A falling unemployment rate signals U.S. economic strength, potentially leading to Fed rate hikes, supporting USD/JPY.<br>- **Series ID: JPNURTOTMISMEI**<br> — **Description**: Japan Unemployment Rate (monthly, from OECD).<br> — **Why It Matters**: Less impactful than U.S. data, but a rising rate could weaken JPY further if it prompts BoJ stimulus.</p><p>- **Series ID: GDP**<br> — **Description**: U.S. Real Gross Domestic Product (quarterly).<br> — **Why It Matters**: Strong U.S. GDP growth bolsters USD, especially if Japan’s GDP (e.g., JPNRGDPEXP) lags.</p><h4><strong>Which Types to Focus On?</strong></h4><p>- **Interest Rates**:<br> — **U.S.**: Federal Funds Rate (FEDFUNDS) for policy, 10-Year Treasury (DGS10) for yield curve and carry trade signals.<br> — **Japan**: 10-Year Government Bond Yield (IRSTCI01JPM156N) as a proxy for BoJ policy impact.<br> — **Why**: The Fed’s rate hikes versus BoJ’s near-zero policy is a classic USD/JPY driver. Long-term yields (DGS10) often lead forex trends.</p><p>- **Inflation**:<br> — **U.S.**: Core CPI (CPILFESL) for Fed signals, PCEPI for policy precision.<br> — **Japan**: Headline CPI (JPNCPIALLMINMEI) to gauge BoJ’s deflation fight.<br> — **Why**: U.S. inflation drives rate expectations, while Japan’s low inflation keeps JPY weak.</p><p>### Practical Approach in Time Series Lab<br>1. **Pull Data**: Use your API key to fetch DEXJPUS, FEDFUNDS, DGS10, IRSTCI01JPM156N, CPIAUCSL, and JPNCPIALLMINMEI.<br>2. **Analyze Differentials**: Plot U.S.-Japan yield and inflation gaps over time alongside USD/JPY.<br>3. **Correlate**: Look for lagged effects (e.g., Fed rate hikes leading USD/JPY by 1–3 months).<br>4. **Monitor Releases**: FRED updates align with official releases (e.g., CPI monthly, Fed rates post-FOMC).</p><p>For USD/JPY, the interplay of U.S. yields (DGS10) and Fed policy (FEDFUNDS) against Japan’s stagnant rates and inflation is your bread and butter. Start with these.</p><h4>What Are Differentials?</h4><p>A differential is just a fancy word for the **difference** between two numbers or values. In forex trading, like with USD/JPY, differentials matter because they show how one country’s economic conditions compare to another’s — and that comparison often drives the exchange rate.</p><p>Imagine you’re comparing two things to see which one “wins” or has more influence. In USD/JPY trading, you’re comparing the U.S. and Japan on key metrics like interest rates or inflation. The differential is the gap between those two metrics. If the gap gets bigger or smaller, it can push the exchange rate (USD/JPY) up or down.</p><h4>Here’s the logic in simple steps:</h4><p>1. Two Countries, Two Values: You take a metric from the U.S. (like interest rates) and the same metric from Japan.<br>2. Subtract Them: U.S. value minus Japan value = the differential.<br>3. Watch the Gap: If the differential grows or shrinks, it signals which currency might get stronger.</p><p><strong>Why Do Differentials Matter for USD/JPY?<br></strong>USD/JPY is the price of one U.S. dollar in Japanese yen. When the U.S. looks “stronger” than Japan economically, the dollar gains value, and USD/JPY goes up (more yen per dollar). Differentials help you measure that strength by comparing key drivers like interest rates and inflation.</p><p><strong>Example 1: Interest Rate Differential</strong><br>- **U.S. Interest Rate**: Say the 10-Year Treasury yield (DGS10 on FRED) is 4%.<br>- **Japan Interest Rate**: Say Japan’s 10-year bond yield (IRSTCI01JPM156N) is 0.5%.<br>- **Differential**: 4% — 0.5% = **3.5%**.</p><p><strong>What does this mean?</strong><br>- A 3.5% differential means U.S. investments pay way more interest than Japanese ones.<br>- Traders like higher returns, so they buy USD (selling JPY), pushing USD/JPY up.<br>- If the U.S. rate rises to 4.5% (differential becomes 4%), the gap widens, and USD/JPY might climb even more.</p><p><strong>Example 2: Inflation Differential</strong><br>- **U.S. Inflation**: Say Core CPI (CPILFESL) shows 3% yearly inflation.<br>- **Japan Inflation**: Say Japan’s CPI (JPNCPIALLMINMEI) is 0.5%.<br>- **Differential**: 3% — 0.5% = **2.5%**.</p><p><strong>What does this mean?</strong><br>- Higher U.S. inflation might force the Fed to raise rates, making USD stronger.<br>- Japan’s low inflation keeps the BoJ relaxed with low rates, weakening JPY.<br>- A growing gap (e.g., U.S. inflation jumps to 4%) could lift USD/JPY.</p><p><strong>How to Use Differentials Logically</strong><br>1. **Get the Numbers**: Pull U.S. and Japan data from FRED (e.g., DGS10 for U.S. yields, IRSTCI01JPM156N for Japan).<br>2. **Subtract**: Calculate the differential (U.S. minus Japan).<br>3. **Track Over Time**: If the differential gets bigger, USD usually strengthens (USD/JPY rises). If it shrinks, JPY might gain (USD/JPY falls).<br>4. **Combine Metrics**: A big interest rate differential plus a big inflation differential = stronger signal for USD/JPY moves.</p><p><strong>Simple Rule of Thumb</strong><br>- **Widening Differential** (U.S. number grows faster than Japan’s): USD/JPY tends to rise.<br>- **Narrowing Differential** (Japan’s number catches up or U.S. slows): USD/JPY tends to fall.</p><p><strong>Real-World Picture</strong><br>Think of it like a tug-of-war:<br>- U.S. pulls with high interest rates and inflation (e.g., 4% yields, 3% CPI).<br>- Japan pulls weakly with low rates and inflation (e.g., 0.5% yields, 0.5% CPI).<br>- The differential (3.5% for rates, 2.5% for inflation) shows the U.S. side winning, so USD/JPY goes up.</p><p>In Time Series Lab, you can plot these differentials (e.g., DGS10 minus IRSTCI01JPM156N) next to USD/JPY (DEXJPUS) and see how they line up. It’s a simple subtraction that reveals a big story!</p><h3>The Role of Inflation Expectations</h3><p>While interest rates and actual inflation (like CPI) reflect current or past conditions, <strong>inflation expectations</strong> add a forward-looking dimension to differentials. They represent what markets, consumers, or economists predict inflation will be in the future. For USD/JPY trading, the differential in inflation expectations between the U.S. and Japan can signal where exchange rates might head before changes show up in actual data.</p><h4>Why Inflation Expectations Matter</h4><ol><li><strong>Leading Indicator</strong>: Expectations often predict future inflation and interest rate shifts. If U.S. inflation is expected to rise faster than Japan’s, markets may anticipate higher U.S. interest rates, strengthening the USD against the JPY.</li><li><strong>Central Bank Impact</strong>: The Fed and BoJ watch inflation expectations closely. Rising U.S. expectations might prompt the Fed to tighten policy, while low or deflationary expectations in Japan could keep the BoJ accommodative, widening the interest rate gap.</li><li><strong>Self-Fulfilling Effect</strong>: If people expect higher inflation, they might raise prices or wages, making it happen. This dynamic amplifies the influence of expectations on economic outcomes.</li></ol><h4>Measuring Inflation Expectations</h4><ul><li><strong>Market-Based</strong>: The difference between nominal and inflation-protected bond yields. For the U.S., FRED offers the <strong>5-Year Breakeven Inflation Rate (T5YIE)</strong> (e.g., 2.5%). Japan’s equivalent may require other sources, like BoJ forecasts or bond yield spreads.</li><li><strong>Surveys</strong>: Consumer or economist polls, like the University of Michigan’s survey for the U.S.</li><li><strong>Central Bank Forecasts</strong>: Fed and BoJ projections shape market views.</li></ul><h4>Example: Inflation Expectations Differential</h4><ul><li><strong>U.S.</strong>: 5-Year Breakeven Inflation (T5YIE) at 2.5%.</li><li><strong>Japan</strong>: Hypothetical equivalent at 0.5%.</li><li><strong>Differential</strong>: 2.5% — 0.5% = <strong>2.0%</strong>.</li></ul><p><strong>What it means</strong>:<br> A 2% gap suggests markets expect U.S. inflation to outpace Japan’s over the next five years. If this differential widens (e.g., U.S. rises to 3%), it could signal tighter Fed policy ahead, boosting USD/JPY. A narrowing gap might hint at Japan’s inflation catching up, potentially weakening USD/JPY.</p><h4>Why It’s Worth Including</h4><p>Inflation expectations enhance your analysis by:</p><ul><li><strong>Anticipating Policy Shifts</strong>: They hint at future interest rate moves before actual inflation changes.</li><li><strong>Adding Depth</strong>: Combining them with actual differentials (e.g., CPI or yields) gives a fuller picture of USD/JPY drivers.</li><li><strong>Market Sensitivity</strong>: Forex traders react to expectations, often moving prices ahead of official data.</li></ul><h4>Options Futures and Other Derivatives</h4><p>For trading and analyzing USD/JPY, futures and options contracts can provide valuable insights and tools, especially for understanding market sentiment, hedging, or speculating on price movements. While these derivatives don’t perfectly predict the spot forex pair (USD/JPY), they are closely associated with it and reflect macroeconomic factors like interest rate expectations, inflation differentials, and risk appetite — key drivers of USD/JPY. Quantitative traders (quants) also use additional derivatives beyond futures and options to model and trade this pair. Below, I’ll outline the relevant contracts from FRED and other sources, their predictive power, and other derivatives quants commonly use.</p><h4>Futures Contracts for USD/JPY</h4><p>Futures contracts are standardized agreements to buy or sell a currency at a set price and date, traded on exchanges like the Chicago Mercantile Exchange (CME). They’re useful for USD/JPY traders because they reflect institutional positioning and expectations tied to U.S. and Japanese monetary policies.</p><p><strong>1. **Japanese Yen Futures (CME: 6J)**</strong><br> — **Description**: Contracts for 12,500,000 JPY deliverable against USD, quoted in USD per JPY (e.g., 0.006800 means 1 JPY = 0.0068 USD, or USD/JPY = 147.06).<br> — **FRED Availability**: Not directly on FRED, but the spot rate (DEXJPUS) mirrors its inverse (JPY per USD). Futures prices align closely with spot adjusted for interest rate differentials.<br> — **Usefulness**: Tracks market expectations of USD/JPY over time (e.g., 1-month, 3-month contracts). A rising futures price (weaker JPY) often signals expectations of U.S. rate hikes or Japanese easing.<br> — **Predictive Power**: Moderate. Futures reflect forward expectations of spot rates adjusted by the U.S.-Japan interest rate differential (via the interest rate parity formula: Forward = Spot × (1 + U.S. rate) / (1 + Japan rate)). Shifts in open interest or volume can hint at big moves.</p><p><strong>2. **Micro Yen Futures (CME: MJY)**</strong><br> — **Description**: Smaller version of 6J, with 1,250,000 JPY per contract.<br> — **Usefulness**: Offers similar insights but with lower capital requirements, appealing to retail traders or smaller quants testing strategies.<br> — **Predictive Power**: Same as 6J but with less liquidity, so it’s noisier.</p><p>- **How to Analyze**: Compare futures prices to DEXJPUS (spot) on FRED. A premium (futures &gt; spot) suggests USD strength; a discount (futures &lt; spot) suggests JPY strength. Pair this with FRED’s FEDFUNDS and DGS10 (U.S. rates) versus Japan’s bond yields (e.g., IRSTCI01JPM156N) to see if differentials match futures pricing.</p><h4>Options Contracts for USD/JPY</h4><p>Options give the right (not obligation) to buy (call) or sell (put) a currency at a strike price by a set date. They’re traded over-the-counter (OTC) or on exchanges like CME, offering insights into volatility and directional bets.</p><p>1. **Options on Japanese Yen Futures (CME: JY Options)**<br> — **Description**: Call and put options tied to the 6J futures contract (12,500,000 JPY). Strikes are set in USD per JPY (e.g., 0.0068, 0.0070).<br> — **FRED Availability**: Not on FRED, but implied volatility and pricing can be inferred from related metrics like spot (DEXJPUS) and rate differentials.<br> — **Usefulness**: <br> — **Implied Volatility (IV)**: High IV signals expected USD/JPY swings, useful for timing trades.<br> — **Skew**: More demand for calls (USD/JPY up) vs. puts (USD/JPY down) shows bullish or bearish sentiment.<br> — **Predictive Power**: Strong for volatility, moderate for direction. A spike in call buying or IV often precedes USD/JPY rallies, especially during U.S. economic releases (e.g., CPI, FOMC).</p><p>2. **OTC Forex Options (USD/JPY Vanilla Options)**<br> — **Description**: Customizable contracts from banks or brokers, typically in USD notional (e.g., $1M USD vs. JPY). Strikes and expiries vary (1 week to 1 year).<br> — **Usefulness**: Risk reversals (call-put premium difference) reveal market bias. A positive risk reversal (calls pricier) suggests USD/JPY upside expectations.<br> — **Predictive Power**: High for short-term sentiment. Quants use 25-delta risk reversals to gauge if traders expect USD strength (e.g., Fed hikes) or JPY safe-haven demand (e.g., risk-off events).</p><p>- **How to Analyze**: Without direct FRED options data, correlate DEXJPUS moves with U.S. inflation (CPIAUCSL) and yields (DGS10). If options IV or skew rises alongside these, it’s a signal. Access CME options data or OTC quotes from brokers for real-time insights.</p><h4>Predictive Power Caveats</h4><p>- **Futures**: Predict spot direction via interest rate expectations but lag intraday forex moves due to lower frequency (daily settlements).<br>- **Options**: Better for volatility and sentiment but require interpreting IV and skew, which aren’t on FRED. Both are more reactive than predictive — pair them with FRED’s macro data (e.g., FEDFUNDS, CPI) for context.</p><h4>Other Important Derivatives Used by Quants for USD/JPY</h4><p>Quants leverage a broader toolkit to model USD/JPY, often combining futures/options with these derivatives for statistical arbitrage, hedging, or volatility trades:</p><p>1. **Currency Forwards**<br> — **Description**: OTC agreements to exchange USD for JPY at a future date, priced via spot and interest rate differentials.<br> — **Usefulness**: Similar to futures but customizable. Quants arbitrage small pricing differences between forwards and CME futures.<br> — **Quant Angle**: Used in carry trade models (borrow JPY, invest in USD assets) when U.S.-Japan rate gaps widen (e.g., DGS10 vs. Japan yields).</p><p>2. **Swaps (Cross-Currency Basis Swaps)**<br> — **Description**: Agreements to swap USD and JPY interest payments over time, reflecting funding cost differences.<br> — **Usefulness**: The basis (premium/discount) signals USD or JPY liquidity stress. A negative basis (USD costly) often boosts USD/JPY.<br> — **Quant Angle**: Models link basis to USD/JPY spot moves, especially during Fed/BoJ policy shifts.</p><p>3. **Exchange-Traded Funds (ETFs)**<br> — **Description**: Currency ETFs like FXY (Invesco CurrencyShares Japanese Yen Trust) track JPY vs. USD.<br> — **Usefulness**: Proxy for USD/JPY exposure. Options on FXY provide additional volatility signals.<br> — **Quant Angle**: High-frequency traders arbitrage ETF prices vs. spot or futures, exploiting mispricings.</p><p>4. **Non-Deliverable Forwards (NDFs)**<br> — **Description**: Cash-settled forwards for currencies with restrictions (not JPY, but related pairs like USD/KRW can influence USD/JPY).<br> — **Usefulness**: Indirectly tied to JPY via Asian currency correlations.<br> — **Quant Angle**: Used in multi-currency models to capture regional flows affecting JPY.</p><p>5. **Volatility Derivatives (e.g., VIX Futures, Custom Vol Swaps)**<br> — **Description**: VIX futures (CBOE) or OTC volatility swaps tied to USD/JPY.<br> — **Usefulness**: JPY is a safe-haven; rising VIX (risk-off) often strengthens JPY (USD/JPY falls).<br> — **Quant Angle**: Pairs VIX moves with USD/JPY options IV for volatility arbitrage.</p><p>### Putting It Together in Time Series Lab<br>- **Futures**: Pull DEXJPUS (spot) from FRED, then source 6J futures prices (CME data) to compare. Overlay FEDFUNDS and Japan yields.<br>- **Options**: Use OTC/CME options data (not FRED) for IV and skew, cross-referencing with FRED’s CPIAUCSL and DGS10 for drivers.<br>- **Other Derivatives**: Forwards and swaps need OTC data, but you can proxy their impact via FRED’s rate and inflation series.</p><p>For USD/JPY, CME’s 6J futures and options are the go-to exchange-traded tools, with predictive power tied to rate differentials and sentiment. Quants layer on forwards, swaps, and volatility trades to exploit inefficiencies. Start with FRED’s macro data, then integrate real-time derivatives for a full picture!</p><h3>Greeks in quantitative trading</h3><p>The Greeks are essential in options trading, risk management, and derivatives analysis, measuring the sensitivity of an option’s price to various underlying factors.</p><p>### **1. Delta (Δ)**</p><p>* **Function:** Measures the rate of change of the option’s price with respect to the underlying asset’s price.<br>* **Meaning:** Expected change in the option price for a unit change in the underlying asset’s price.<br>* **Use Cases:**<br> + **Hedging:** Used to delta-hedge a portfolio, aiming to neutralize the portfolio’s sensitivity to the underlying asset’s price movements.<br> + **Trading Strategy:** Determines the number of options needed to replicate the exposure of the underlying asset.</p><p>### **2. Gamma (Γ)**</p><p>* **Function:** Measures the rate of change of the option’s delta with respect to the underlying asset’s price.<br>* **Meaning:** Indicates how much the option’s delta is expected to change when the underlying asset’s price changes.<br>* **Use Cases:**<br> + **VaR (Value-at-Risk) Calculations:** Helps in understanding potential portfolio volatility.<br> + **Gamma Trading/Gamma Hedging:** Traders seek to buy options with high gamma to profit from increased volatility or hedge against significant price movements.</p><p>### **3. Vega (ν)**</p><p>* **Function:** Measures the rate of change of the option’s price with respect to the volatility of the underlying asset.<br>* **Meaning:** Reflects the option’s sensitivity to changes in the underlying asset’s volatility.<br>* **Use Cases:**<br> + **Volatility Trading:** Investors buy or sell options based on expected changes in volatility.<br> + **Portfolio Management:** Helps in diversifying portfolios by managing volatility risk.</p><p>### **4. Theta (θ)**</p><p>* **Function:** Measures the rate of change of the option’s price with respect to time.<br>* **Meaning:** Represents the decrease in the option’s value due to the passage of time.<br>* **Use Cases:**<br> + **Options Trading Strategies:** Traders selling (writing) options benefit from theta, as the option’s value decreases over time if other factors remain constant.<br> + **Investment Decisions:** Helps investors decide whether to hold or close positions nearing expiration.</p><p>### **5. Rho (ρ)**</p><p>* **Function:** Measures the rate of change of the option’s price with respect to the interest rate.<br>* **Meaning:** Shows the option’s sensitivity to changes in the risk-free interest rate.<br>* **Use Cases:**<br> + **Interest Rate Environments:** More relevant in environments where interest rates are expected to fluctuate significantly.<br> + **Long-term Options:** More impactful for options with longer maturities.</p><p>### **6. Lambda (λ) — Elasticity or Leverage**</p><p>* **Function:** Measures the percentage change in the option’s price for a percentage change in the underlying asset’s price.<br>* **Meaning:** Demonstrates the leverage of the option compared to the underlying asset.<br>* **Use Cases:**<br> + **Comparative Analysis:** Useful for comparing leverage across different options or strategies.<br> + **Risk Management:** Highlights the potential magnified gains or losses in options trading.</p><p>### **7. Epsilon (not commonly used with the first 5 Greeks but sometimes referenced in specific contexts)**</p><p>* **Function/ Meaning/ Use Cases:** Not standard, but in some texts or software, it may refer to the change in option price with respect to the underlying dividend yield. Its use is more niche compared to the primary Greeks.</p><p>### Additional, Less Common Greeks (for completeness, though not universally standardized or as frequently discussed):</p><p>- **Vanna:** The rate of change of delta with respect to volatility. Useful for analyzing the volatility’s impact on delta-hedging.<br> <br>- **Charm (ΔGamma):** The rate of change of gamma with respect to the underlying asset’s price. Helps in understanding how the sensitivity of delta itself changes.</p><p>- **Vomma (DvegaDvol):** Measures the sensitivity of vega to the underlying asset’s volatility. Important for assessing volatility trading strategies’ risks.</p><p>- **Veta (DvegaDtime):** Reflects how vega changes over time. Useful for longer-dated options or in periods of changing volatility expectations.</p><p>- **Speed (DgammaDspot):** The rate of change of gamma with respect to the underlying asset’s price, more commonly referenced in deeper discussions of options behavior.</p><p>### Simplified Use Case Summary:</p><ul><li>**Trading Strategies:** Delta (hedging), Gamma (volatility), Vega (volatility trading), Theta (time decay)<br>- **Risk Management:** All Greeks, with a focus on Delta, Gamma, and Vega for immediate risk, and Theta for time-sensitive risk.<br>- **Portfolio Management:** Vega for volatility diversification, Rho for interest rate sensitivity, and Lambda for leverage assessments.<br>- **Investment Decisions:** Theta for managing time decay, and the other Greeks contextually based on market expectations.</li></ul><h3>Analytical Workflow</h3><p>Below is a comprehensive, step-by-step quantitative trader’s analytical workflow for trading USD/JPY. This workflow integrates economic data from the Federal Reserve Economic Data (FRED), derivatives insights, and quantitative techniques to ensure your trading decisions are well-informed and data-driven. Each step is designed to build on the previous one, creating a reliable and clear process for analyzing and trading USD/JPY effectively.</p><h4>Quantitative Trader’s Analytical Workflow for USD/JPY</h4><h4>Step 1: Data Collection and Setup</h4><p>**Objective**: Gather the necessary data to analyze USD/JPY comprehensively.</p><p>- **FRED Data** (accessible via FRED API):<br> — **DEXJPUS**: Spot USD/JPY exchange rate (daily).<br> — **FEDFUNDS**: U.S. Federal Funds Rate.<br> — **DGS10**: U.S. 10-Year Treasury Yield.<br> — **IRSTCI01JPM156N**: Japan 10-Year Government Bond Yield.<br> — **CPIAUCSL**: U.S. Consumer Price Index (CPI).<br> — **JPNCPIALLMINMEI**: Japan Consumer Price Index (CPI).<br> — **UNRATE**: U.S. Unemployment Rate.<br> — **JPNURTOTMISMEI**: Japan Unemployment Rate.<br> — **GDP**: U.S. Gross Domestic Product (quarterly).<br> — **JPNRGDPEXP**: Japan Gross Domestic Product (quarterly).</p><p>- **Derivatives Data** (sourced from brokers, CME, or financial platforms):<br> — **Japanese Yen Futures (6J)**: Futures prices, open interest, and volume.<br> — **Options on Japanese Yen Futures**: Implied volatility and skew.<br> — **OTC Forex Options**: Risk reversals for USD/JPY to gauge market bias.</p><p><strong>- **Tools**:</strong><br> — Use software like Time Series Lab with your FRED API key to fetch and organize FRED data.<br> — Supplement with real-time derivatives data from platforms like Bloomberg, Reuters, or your broker.</p><h4><strong>Step 2: Calculate Key Differentials</strong></h4><p>**Objective**: Compute economic differentials to assess the relative strength of the U.S. and Japanese economies.</p><p>- **Interest Rate Differential**:<br> — Formula: `DGS10 — IRSTCI01JPM156N`.<br> — Interpretation: A widening differential favors USD strength (higher U.S. yields relative to Japan).</p><p>- **Inflation Differential**:<br> — Formula: `CPIAUCSL — JPNCPIALLMINMEI` (year-over-year change).<br> — Interpretation: Higher U.S. inflation relative to Japan may pressure USD/JPY upward.</p><p>- **Unemployment Differential**:<br> — Formula: `UNRATE — JPNURTOTMISMEI`.<br> — Interpretation: Lower U.S. unemployment relative to Japan signals U.S. economic strength.</p><p>- **GDP Growth Differential**:<br> — Formula: Difference in quarterly GDP growth rates (`GDP` vs. `JPNRGDPEXP`).<br> — Interpretation: Faster U.S. GDP growth supports a stronger USD.</p><p>- **Action**: Track these differentials over time (daily, weekly, monthly) to identify trends.</p><h4><strong>Step 3: Analyze Derivatives for Sentiment and Volatility</strong></h4><p>**Objective**: Use futures and options data to understand market expectations and potential price movements.</p><p>- **Futures Analysis**:<br> — Compare **6J futures prices** to **DEXJPUS spot rates**.<br> — Insight: Futures trading at a premium suggest expectations of USD appreciation; a discount suggests JPY strength.<br> — Monitor **open interest** and **volume** for shifts in market positioning.</p><p>- **Options Analysis**:<br> — Track **implied volatility (IV)** for USD/JPY options.<br> — High IV signals expected price swings; low IV suggests stability.<br> — Analyze **risk reversals** (difference in IV between calls and puts).<br> — Positive risk reversal (favoring calls) indicates bullish USD sentiment; negative suggests bearish sentiment.</p><p>- **Optional**: If accessible, monitor **cross-currency basis swaps** for signs of liquidity stress impacting USD/JPY.</p><h4><strong>Step 4: Model Relationships and Correlations</strong></h4><p>**Objective**: Use statistical tools to identify predictive patterns and relationships.</p><p>- **Time Series Analysis**:<br> — Plot **DEXJPUS** against interest rate, inflation, and GDP differentials.<br> — Introduce lagged variables (e.g., differential changes from 1–3 months prior) to test if they lead USD/JPY movements.</p><p>- **Correlation Analysis**:<br> — Calculate correlation coefficients between **DEXJPUS** and each differential.<br> — Focus on strong correlations (e.g., &gt;0.7 or &lt;-0.7) to identify key drivers.</p><p>- **Regression Models**:<br> — Build a linear regression with **DEXJPUS** as the dependent variable and differentials as independent variables.<br> — Use the model to forecast USD/JPY based on projected differential changes.</p><p>- **Tools**: Use Python (pandas, statsmodels), R, or Excel for calculations.</p><h4><strong>Step 5: Monitor Economic Releases and Events</strong></h4><p>**Objective**: Stay ahead of market-moving events and data releases.</p><p>- **Key U.S. Events**:<br> — **FOMC Meetings**: Impact on **FEDFUNDS** and yield curve.<br> — **CPI/PCE Reports**: Inflation updates.<br> — **Non-Farm Payrolls**: Employment strength affecting USD.</p><p>- **Key Japanese Events**:<br> — **Bank of Japan (BoJ) Meetings**: Policy shifts impacting JPY.<br> — **CPI and GDP Releases**: Economic health indicators.</p><p>- **Global Factors**:<br> — Watch for geopolitical tensions or risk-off events (JPY often strengthens as a safe haven).</p><p>- **Action**: Maintain an economic calendar and adjust positions ahead of high-impact releases.</p><h4><strong>Step 6: Implement Trading Strategies</strong></h4><p>**Objective**: Develop actionable trading strategies based on your analysis.</p><p>- **Carry Trade Strategy**:<br> — **Condition**: Wide and stable interest rate differential favoring USD.<br> — **Action**: Go long USD/JPY to earn the yield spread.<br> — **Watch**: Narrowing differentials or rising JPY volatility as exit signals.</p><p>- **Momentum Trading**:<br> — **Tools**: Use moving averages (e.g., 50-day vs. 200-day) on **DEXJPUS**.<br> — **Action**: Enter trades in the direction of confirmed trends, supported by differential movements.</p><p>- **Volatility Trading**:<br> — **Condition**: High implied volatility in options.<br> — **Action**: Sell options to capture premium or use straddles/strangles before major events.</p><h4><strong>Step 7: Risk Management and Evaluation</strong></h4><p>**Objective**: Protect capital and assess strategy performance.</p><p>- **Position Sizing**:<br> — Use fixed fractional sizing (e.g., risk 1–2% of capital per trade) or the Kelly criterion.<br> — Adjust based on volatility (e.g., Average True Range).</p><p>- **Stop Losses and Take Profits**:<br> — Set stop losses at technical levels (support/resistance) or volatility-based thresholds.<br> — Target take profits at a minimum 2:1 risk-reward ratio.</p><p>- **Performance Metrics**:<br> — Track **win rate**, **average win/loss ratio**, and **Sharpe ratio**.<br> — Backtest strategies using historical data and refine based on results.</p><h4><strong>Step 8: Continuous Learning and Adaptation</strong></h4><p>**Objective**: Refine your approach as markets evolve.</p><p>- **Market Research**:<br> — Follow central bank statements, financial news (e.g., Bloomberg, Reuters), and trader forums.</p><p>- **Model Updates**:<br> — Re-run models with fresh data quarterly.<br> — Test additional variables (e.g., equity market correlations) for improved accuracy.</p><p>- **Technology**:<br> — Explore machine learning (e.g., random forests, neural networks) for pattern recognition.<br> — Automate data retrieval and analysis with scripts (e.g., Python).</p><p><strong>Conclusion</strong><br>This workflow provides a structured, data-driven approach to trading USD/JPY. By combining FRED economic data, derivatives insights, and quantitative analysis, you can make informed decisions with a clear process. Start with small positions to test your strategies, monitor performance, and adapt as needed. Trading success hinges on discipline, risk management, and continuous improvement — stick to the workflow and refine it over time to suit your style and market conditions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=707018c20241" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Hands with Docker Compose as Desktop App]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/open-hands-with-docker-compose-as-desktop-app-0c628b7415e0?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c628b7415e0</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[ide]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Mon, 10 Mar 2025 20:07:05 GMT</pubDate>
            <atom:updated>2025-03-11T20:00:50.882Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Open Hands ?</h3><p>Open Hands AI is a useful tool that allows you to connect with and leverage various Large Language Models through an intuitive interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*tOt9ZGoJNjlg0BYsKxMXjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*EnIeVG2jIaNKb0AV.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_q7NT0Lo00h13kdO.png" /></figure><p><strong>It can do anything a human developer can: </strong><br>- modify code <br>- run commands<br>- browse the web<br>- call APIs<br>- interact with your Github account repository<br>- even copy code snippets from StackOverflow</p><p>All of this while ensuring privacy, as your API keys and data remain entirely local. For complete privacy, you can even integrate your locally hosted LLMs from Ollama within OpenHands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*CcB-TQIr3DG-qrZ9lf_35Q.png" /></figure><p><strong>Prerequisites</strong></p><p>Before we begin, make sure you have:</p><ul><li>Basic familiarity with command line interfaces</li><li>Docker Desktop installed</li><li>Your <strong>API key</strong> ready for your preferred LLM (like OpenAI , Anthropic , Deepseek , Grok etc.)</li></ul><h3>Let&#39;s dive in!</h3><h3><strong>Step 1</strong></h3><p>Go to your terminal and create an empty directory in your path and call it whatever you like. I’ll call it open-hands-app for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uKnudjoziJ7TT-fy9aTSWQ.png" /></figure><p>after that enter the directory with <strong><em>cd open-hands-app</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*g_moFv4_NRYYa31OKRV1Yg.png" /></figure><p>Great Job;)</p><h3><strong>Step 2</strong></h3><p>Since we want to run OpenHands via Docker Compose next step is to write a docker-compose.yaml file.</p><p>If you are using VS Code , you can easily navigate to the folder we previously made. Following the flow , create and edit the docker-compose.yaml file via the IDE interface (recommended).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xnG4rUpDxS-q3XLHC2tpA.png" /></figure><p>Otherwise, you can easily edit the yaml file directly in your terminal as well:</p><ul><li>Vim editor with vi docker-compose.yaml</li><li>GNU editor with nano docker-compose.yaml</li></ul><p>Both will do the job. Just make sure to format the yaml file with proper indentations.</p><pre><br>services:<br>    all-hands-ai:<br>        image: &#39;docker.all-hands.dev/all-hands-ai/openhands:0.28&#39;<br>        container_name: openhands-app<br>        extra_hosts:<br>            - &#39;host.docker.internal:host-gateway&#39;<br>        ports:<br>            - &#39;3000:3000&#39;<br>        volumes:<br>            - &#39;.openhands-state:/.openhands-state&#39;<br>            - &#39;/var/run/docker.sock:/var/run/docker.sock&#39;<br>        environment:<br>            - LOG_ALL_EVENTS=true<br>            - &#39;SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik&#39;<br>          #  - LLM_API_KEY: &quot;NA&quot; # You can use a self hosted LLM if you like<br>          #  - LLM_BASE_URL: &quot;Ollama address:11434&quot; # You can use a self hosted LLM if you like<br>        pull_policy: always<br>        tty: true<br>        stdin_open: true</pre><h3>Step 3</h3><p>Save the yaml file, go back to your terminal and type:</p><pre>docker-compose up</pre><p>and see the magic happen.</p><p>Take your time though , this might take a while due to its size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BXazR-__GCuBw__lS4yug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*Qoh4rCunLYr0mboL.jpg" /></figure><h3>Step 4</h3><p>After starting the docker compose i.e. containers, access the application through your browser:</p><ul><li>Check Docker Desktop dashboard for the mapped port, or find it in your terminal output or docker logs.</li><li>Open your browser and navigate to:</li></ul><pre>http://localhost:3000</pre><p>Your application should be running locally now.</p><p>When you launch OpenHands, you’ll be prompted to choose your preferred LLM provider and model. Simply enter your API key, which you’ve obtained directly from platforms like OpenAI, DeepSeek, and others.</p><p>Based on recent benchmarks its recommended to use Anthropic Claude 3.7 Sonnet model for the best performance . For more details check link below:</p><p><a href="https://www.swebench.com/#test">SWE-bench</a></p><h3><strong>Final Step (Optional but fun)</strong></h3><p>Microsoft Edge browser offers a nice feature that lets you transform any website into a desktop app. If you have your OpenHands app running in Docker in the background and want quick access directly from your Dock with a nice icon, here’s what to do:</p><p>With your OpenHands app running on localhost:3000 (or whatever port you’re using) in Edge browser, click the three dots (⋯) in the top-right corner. Navigate to “Apps” in the dropdown menu, then select “Install this site as an app.”</p><p>This creates a standalone Edge app that you can pin to your dock panel. While it technically still runs in a browser, it provides the clean experience of a desktop application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d0ZrqoF4WGC8AbkVVMQCTw.png" /></figure><p><strong>Just remember</strong>: if Docker stops or your containers go down for unknown reason, make sure to restart the ones dedicated to your OpenHands App with docker compose up command.</p><p>Once running, you&#39;ll always be able to access OpenHands with a simple click on the icon in your dock panel!</p><p><em>Thank you for your attention! Enjoy coding with OpenHands.<br>Till next one!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c628b7415e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Searching Algorithms - finding optimal trading opportunities across universe of selected assets]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/searching-algorithms-finding-optimal-trading-opportunities-across-universe-of-selected-assets-41ff2ac9ad8e?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/41ff2ac9ad8e</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Mon, 29 Jul 2024 12:37:51 GMT</pubDate>
            <atom:updated>2024-07-30T14:00:08.334Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In my view, the principal factor contributing to losses among traders is the inability to rapidly search for and identify optimal trading opportunities among the vast array of available market instruments. The human brain is inherently limited in its capacity to process and evaluate thousands, or even tens of thousands, of financial instruments to pinpoint the ideal environment where a given strategy, model, or method might perform successfully in a given moment.</em></p><p><em>Given the complexity and scale of today’s financial markets, this challenge underscores the need for advanced analytical tools and techniques capable of efficiently scanning and assessing large volumes of data. The traditional manual approach is increasingly inadequate in navigating the intricate landscape of financial assets. As such, there is a pressing need to enhance our understanding and capabilities in developing and deploying sophisticated systems that can automate and optimize this search process.</em></p><p><em>Explore this topic in greater depth to uncover effective strategies and technologies that can address these challenges and improve our trading objectives.</em></p><h4>What is searching?</h4><p><em>Searching is the fundamental process of </em><strong><em>locating a specific element or item within a collection of data. </em></strong><em>In searching, there is always a specific </em><strong><em>target element or item</em></strong><em> that you want </em><strong><em>to find within the data collection.</em></strong><em> This target could be a value, a record, a key, or any other data entity of interest.</em></p><p><strong><em>The search space</em></strong><em> on the other hand refers to the entire collection of data within which you are looking for the target element. Depending on the data structure used, the search space may vary in size and organization.</em></p><p><em>In our case we want to search for a specific behavior of price data over time or under other domains which given model or anything , can describe the behaviour and provide some predictive value for our benefit.</em></p><p>So what is a behavior of time series then?</p><p><strong><em>Behavior of a Time Series</em></strong><em> refers to the domain specific patterns, trends, cycles , forces &amp; structures that characterize the data observed sequentially over time.</em></p><p><em>It encompasses several fundamental aspects that describe how the time series evolves and changes. Its nature (characteristics) is essential for understanding &amp; effective modeling.</em></p><h3>Keep it rollin…</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*rDnLUYvvcGrEwDBg.gif" /></figure><h4><strong><em>What would be our Objective?</em></strong></h4><p><em>We want to identify the best environment or asset at a specific time T where our model can make the most accurate and effective decisions (of opening a short term directional trade for instance).</em></p><h4><strong><em>Details:</em></strong></h4><p><strong><em>Model Parameters (θ , … , …)<br></em></strong><em>are the random variables that our model uses to make decisions.</em></p><p><strong><em>Model Assumptions (A(θ)) <br></em></strong><em>are the conditions or requirements that need to be met for the model to make decisions.</em></p><p><strong><em>Database/Dictionary of Assets (D)<br></em></strong><em>This is a collection of unique environments (assets’ prices over specified time window).</em></p><p><em>We can search/iterate/scan (call it whatever) thru the collection independently, where each asset is a unique object of our collection -&gt; a ∈ D. Additionally we could consider other variables associated with each unique asset but lets not get stuck on it now.</em></p><p><strong>Objective Function (f)<br></strong><em>This function measures how well our model performs in each environment/asset.</em></p><p><strong><em>Meta Objective Function — M(f)</em></strong><em><br>We could think about a secondary objective function which we could call </em><strong><em>Meta Objective Function</em></strong><em>.</em></p><p><strong><em>The Meta Objective Function</em></strong><em>, on the other hand, would provide insight into </em><strong><em>which characteristics</em></strong><em> or features of the (environment) assets </em><strong><em>lead to the best</em></strong><em> performance of your model. In short helping us to identify the “</em><strong><em>favourable characteristics”</em></strong><em>.</em></p><p><strong><em>For instance <br></em></strong><em>If your primary Objective Function is accuracy, the Meta Objective Function could be something that assesses the impact of different features (e.g., data quality, feature distribution, environment conditions) on accuracy.</em></p><p><em>Our primary</em><strong><em> Objective Function’s</em></strong><em> final destination would be </em><strong><em>High Information Ratio</em></strong><em> and our </em><strong><em>Meta Objective Function</em></strong><em> might be finding “</em><strong><em>favourable characteristics</em></strong><em>” that lead to High Information Ration.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/0*XE5vZ7jakWSQGXO6" /></figure><p>Assume you have already run a search algorithm on various combinations of features , strategies and models parameters and you have a dataset with the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*XYOFfQTlLV5aw_4GKiVIIw.png" /></figure><h4><strong>The meta objective function can be formalized as follows:</strong></h4><p><em>Meta Objective Function= Maximize(IR(F2,F5,S1,{P1,P2})</em></p><p><em>Where IR represents the </em><strong><em>Information Ratio</em></strong><em> as a function of the favorable characteristics which can be then ranked in descending order.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41ff2ac9ad8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What do asset prices and space rockets have in common?]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/what-do-asset-prices-and-space-rockets-have-in-common-4330c56f062b?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/4330c56f062b</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 01:36:36 GMT</pubDate>
            <atom:updated>2023-07-22T21:18:03.617Z</atom:updated>
            <content:encoded><![CDATA[<p><em>It seems like an interesting analogy to compare the evolving price data of an asset to a rocket in space. Both scenarios involve dealing with complex systems where predictions are crucial, and models play a significant role in understanding and navigating them.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*cgOM9w7hQ0s5etUl.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/0*_uusZOA401g3GHzE" /></figure><p><em>In the context of financial markets, predicting asset prices is indeed challenging, and it shares some similarities with space exploration. Here are some key points to consider in this analogy:</em></p><p><strong><em>1. Complex Systems: </em></strong><em>Both space and financial markets are highly complex systems with numerous factors influencing their behavior. In space, gravitational forces, celestial bodies, and other cosmic phenomena interact to determine the path of a rocket. In financial markets, factors like economic indicators, company performance, geopolitical events, and investor sentiment influence asset prices.</em></p><p><strong><em>2. Unobservable Universe:</em></strong><em> In space, there are regions of the universe that are unobservable due to their distance or limitations in technology. Similarly, in financial markets, the future is unobservable, and the prices that will unfold are unknown. Predictions are made based on historical data, patterns, and models, but there’s always uncertainty.</em></p><p><strong><em>3. Using Models for Navigation:</em></strong><em> Just as space missions use sophisticated models and simulations to navigate and make predictions about celestial bodies’ positions, financial analysts and traders use various models to forecast asset prices. These models can range from simple moving averages to complex machine learning algorithms.</em></p><p><strong><em>4. External Factors and Approximations: </em></strong><em>In space exploration, accurate predictions require understanding and accounting for various external factors like gravity, radiation, and more. Similarly, in finance, external factors such as interest rates, market sentiment, and global events need to be considered when making predictions.</em></p><p><strong><em>5. Extrapolation and Risks: </em></strong><em>Extrapolating historical price data to predict future prices involves inherent risks. Just like a rocket’s trajectory might deviate due to unforeseen events in space, financial markets can be affected by unexpected news, market shocks, or changes in investor behavior.</em></p><p><strong><em>6. Adaptation and Agility:</em></strong><em> Both space missions and trading require adaptability and quick decision-making. Spacecraft must adjust their course based on real-time information, and traders must respond to market movements and news etc.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4330c56f062b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning algorithms are function replication systems]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/machine-learning-algorithms-are-function-replication-systems-a03130d4df1e?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/a03130d4df1e</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Sun, 16 Jul 2023 18:25:23 GMT</pubDate>
            <atom:updated>2023-07-20T01:38:26.686Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Machine learning algorithms can be considered as function replication systems to some extent. The primary goal of many machine learning algorithms is to learn a function that maps input data to desired output or predictions. By training on a dataset, these algorithms aim to approximate the underlying function or relationship between the inputs and outputs.</em></p><p><em>The process typically involves feeding the algorithm with a set of labeled training data, consisting of input examples and their corresponding correct outputs. The algorithm then learns from these examples and adjusts its internal parameters to minimize the difference between its predicted outputs and the true outputs. This adjustment is often done through an optimization process, such as gradient descent.</em></p><p><em>Once the training is complete, the learned model can replicate the function or relationship observed in the training data. This means that when new, unseen input data is presented to the model, it can generate predictions or outputs based on what it has learned during training.</em></p><h4>The Universal Approximation Theorems</h4><blockquote>Cybenko et al.</blockquote><p><em>provide theoretical support for the remarkable approximation capabilities of neural networks.</em></p><p><em>They suggest that, in theory, a single hidden layer with a large enough number of neurons can approximate any continuous function, regardless of its complexity.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a03130d4df1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quant Trader Objectives]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/quant-trader-objectives-8df041c02da8?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/8df041c02da8</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Sun, 16 Jul 2023 18:15:08 GMT</pubDate>
            <atom:updated>2023-07-16T18:22:07.268Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>A quant trader has two primary objectives:</em></strong></p><h4><em>1.</em></h4><ul><li><em>First, they aim to develop a statistically significant model that can accurately predict future returns of a group of assets using historical data.</em></li><li><em>The success of achieving this objective is influenced by the methods they employ and the data they choose to analyze.</em></li><li><em>Ultimately, the desired outcome is a predictive function, often referred to as “alpha,” which represents the expected future returns of the assets in question.</em></li><li><em>This alpha function serves as a valuable tool for making informed investment decisions and maximizing profitability in quantitative trading.</em></li></ul><h4><em>2.</em></h4><ul><li><em>The second objective of a quant trader is to utilize the obtained information effectively to generate profitable investments in the markets.</em></li><li><em>However, they are aware of several factors that affect their trading outcomes. These factors include the imprecision of their information and the influence of unpredictable market forces on returns.</em></li><li><em>Additionally, risk controls and operational limits impact their trading strategy, while transaction costs and other frictions can impact their overall profitability.</em></li><li><strong><em>To address these challenges, </em></strong><em>the quant trader aims to </em><strong><em>develop a policy function</em></strong><em> that guides their position-taking decisions based on the alpha function, ultimately maximizing their investment returns while considering the aforementioned factors.</em></li></ul><h4><strong><em>Designing policy function framework</em></strong></h4><p><em>also known as a </em><strong><em>holding function</em></strong><em>, is a mathematical expression, algorithm, or set of rules that takes into account a trader’s alpha (expected future returns), risk framework, and current position to provide a recommended new position.</em></p><p><em>In simpler terms, it tells the trader what position they should hold based on their information about future returns.</em></p><p><em>The difference between the output of the holding function and the current position indicates the trade that should be executed at the next decision point.</em></p><p><strong><em>This policy function serves as a guide for making informed trading decisions and adjusting positions in response to changing market conditions.</em></strong></p><blockquote><em>The functional form of the holding function is chosen to maximize some objective computed from the expected future positions of the strategy.</em></blockquote><blockquote><em>We are already familiar with one such function, the Markowitz function from Mean-Variance Optimization</em></blockquote><h4><em>Information Sets</em></h4><p><a href="https://en.wikipedia.org/wiki/Information_set_(game_theory)#:~:text=The%20information%20set%20is%20the,concept%20in%20non%2Dperfect%20games.">Information set (game theory) - Wikipedia</a></p><p><em>Many quant traders talk about the “information set” in vague terms. To analyze trading strategies we need to make this concept concrete. We need to itemize the elements of the future information set that are specific to a trading decision. And understand that they are stochastic in nature.</em></p><ul><li><em>The</em><strong><em> process of solving for the holding function</em></strong><em> that is, in some way, optimal is therefore an exercise in </em><strong><em>the discipline of Stochastic Programming.</em></strong></li><li><em>This is the mathematical analysis of decision making under uncertainty.</em></li><li><em>Adapted to the specific needs of systematic traders and the environments they are in.</em></li></ul><ul><li><a href="https://en.wikipedia.org/wiki/Stochastic_programming">Stochastic programming - Wikipedia</a></li><li><a href="https://pyomo.readthedocs.io/en/stable/modeling_extensions/stochastic_programming.html">Stochastic Programming in Pyomo - Pyomo 6.6.1 documentation</a></li><li><a href="https://pysp.readthedocs.io/en/latest/pysp.html">Stochastic Programming - PySP 6.0.0 documentation</a></li></ul><h4><em>There are two important functions of a information set that are part of our daily life as traders.</em></h4><p><em>1. </em><strong><em>The Metric Function</em></strong><em>, m(It), selects the information we measure the world by and presents it to us. This is something you get to chose. It will be things like the prices of securities or their returns.</em></p><p><em>2. </em><strong><em>The Alpha</em></strong><em>, α(Is), represents measurements we make from a prior information set that are informative of future values of the metrics of choice</em></p><h4><em>When considering information sets and useful alphas, it is crucial to ensure that the alphas we create are unbiased and capable of providing insights into future expectations.</em></h4><p><em>In other words, we want our alpha, denoted as α(Is), to satisfy the condition α(Is) = E[m(It)| Is] for t &gt; s.</em></p><p><strong><em>One common approach to achieve this is by utilizing the linear additive noise framework</em></strong><em>, which is expressed as m(It) = α(Is) + εt. Here, εt represents a random innovation with a mean of zero.</em></p><ul><li><em>By incorporating this framework, the alpha α(Is) captures the expected future returns based on the available information set Is.</em></li><li><em>The random innovation term εt introduces the element of uncertainty or unpredictability into the alpha calculation.</em></li><li><strong><em>This framework allows us to create alphas that are not only informative but also capable of providing unbiased predictions about future returns</em></strong><em>.</em></li></ul><h4><em>Solving for optimal trading strategies involves performing high-dimensional multivariate integrals over the space of stochastic alphas, holdings and returns.</em></h4><ul><li><em>Specific solutions are possible, but the problem is analytically stiff in general.</em></li><li><em>The Universal Approximation Theorem holds out the potential of using modern Machine Learning methods such as Deep Learning to shortcut the process of numerical integration necessary to solve trading problems.</em></li><li><em>The necessary training set is the set of positions that are optimal ex post given a performance scoring function.</em></li><li><em>Solutions will require the representation of both linear and radial discriminant functions by networks.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8df041c02da8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing Time Series Forecasting: Leveraging Machine Learning and Search Algorithms to Outperform…]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/enhancing-time-series-forecasting-leveraging-machine-learning-and-search-algorithms-to-outperform-2bb69a95f490?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bb69a95f490</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Sun, 16 Jul 2023 16:08:36 GMT</pubDate>
            <atom:updated>2023-07-17T11:26:53.080Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Enhancing Time Series Forecasting: Leveraging Machine Learning and Search Algorithms to Outperform Baseline Models</em></h4><p><strong><em>Introduction:</em></strong><em><br>Time series forecasting is a crucial task in numerous domains, where traditional approaches like ARIMA (AutoRegressive Integrated Moving Average) serve as baseline models. However, with the advancements in machine learning and search algorithms, there is an opportunity to enhance forecasting accuracy beyond these traditional methods.</em></p><p><em>In this article, we explore the concept of leveraging machine learning algorithms and search techniques to surpass the forecasting capabilities of baseline models like ARIMA(0,1,1) which we can express as Integrated Moving Avarage IMA(1,1).</em></p><p><strong><em>Why Integrated Moving Average ?<br></em></strong><em>Simple motivation to favour an IMA(1,1) model as a benchmark is because it can be written as a simple random walk forecast update but now where past forecast errors are accommodated, where still the prediction interval can simply be computed (Chatfield, </em><a href="https://www.tandfonline.com/doi/full/10.1080/13504851.2019.1686115#"><em>Citation1993</em></a><em>).</em></p><p><em>The quality of a forecast is often measured in terms of “forecasting skill,” defined to be</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*zJgh8P-akUBjSjk-.png" /></figure><p><em>and the baseline should be the IMA(1,1).</em></p><ul><li><a href="https://en.wikipedia.org/wiki/Mean_squared_error">Mean squared error - Wikipedia</a></li><li><a href="https://en.wikipedia.org/wiki/Forecast_skill">Forecast skill - Wikipedia</a></li></ul><p><strong><em>Defining the Objective:<br></em></strong><em>Our primary goal is to discover a machine learning algorithm that outperforms the baseline models in terms of forecasting accuracy. To achieve this, we integrate ML and or search algorithms that assist in identifying optimal hyperparameters and model configurations, further enhancing the forecasting power or helping us to pick the right model out of a </em><strong><em>SET </em></strong><em>of selected models.</em></p><p><strong><em>2. Dataset Preparation:<br></em></strong><em>To ensure accurate forecasting, we begin by preparing the dataset in a suitable format. Cleaning the data, handling missing values, outliers, and noise (denoising ) are essential steps in creating a robust foundation for training and evaluating the machine learning models.</em></p><p><strong><em>3. Feature Engineering:<br></em></strong><em>Feature engineering plays a pivotal role in extracting valuable insights from time series data. By identifying relevant features such as lagged values, rolling statistics, or seasonal components etc. , we capture the underlying patterns and dynamics of the time series. These engineered features serve as inputs to both the machine learning algorithms and search algorithms, facilitating comprehensive analysis.</em></p><p><strong><em>4. Model Selection:<br></em></strong><em>To surpass the baseline models, we assemble a diverse set of machine learning algorithms, ranging from regression algorithms like linear regression and decision trees to advanced techniques like random forests, gradient boosting, support vector machines, and neural networks or more sophicticated models.</em></p><p><strong><em>This wide selection ensures a thorough exploration of possibilities.</em></strong></p><p><strong><em>5. Training and Evaluation with Search Algorithms:<br></em></strong><em>In addition to training the machine learning models, we incorporate search algorithms to optimize the hyperparameters and model configurations. Search algorithms like grid search, random search, or Bayesian optimization systematically explore the hyperparameter space, enabling us to discover the best combinations that maximize forecasting performance. The models are then evaluated using appropriate metrics like mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), or others tailored to the specific requirements.</em></p><p><strong><em>6. Fine-Tuning and Iteration:<br></em></strong><em>If the initial search does not yield satisfactory results, further fine-tuning can be performed. This may involve refining the search space, adjusting search parameters, or exploring ensemble techniques to improve performance. The iterative process allows for continuous adjustments based on insights gained during evaluation.</em></p><p><strong><em>7. Selecting the Best Model:<br></em></strong><em>Following thorough evaluation and potential fine-tuning, we select the machine learning algorithm that demonstrates superior forecasting power compared to the baseline models. This algorithm, powered by the integration of machine learning and search algorithms, becomes the chosen approach for time series forecasting in the given context.</em></p><h4>Evaluating Models</h4><p><a href="https://en.wikipedia.org/wiki/Taylor_diagram">Taylor diagram - Wikipedia</a></p><p><strong>Taylor diagrams <br></strong><em>are mathematical diagrams created by Karl E. Taylor in 1994 to compare different models and determine which one best represents a system or phenomenon. These diagrams are useful for evaluating models used in climate and environmental studies, but can also be applied to other areas like fusion energy. They quantify the similarity between modeled and observed behavior using three statistics: Pearson correlation coefficient, root-mean-square error (RMSE), and standard deviation. Taylor diagrams provide a visual representation of how well models match reality.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*gsq_B5hgDR5bt2qQ.png" /></figure><p><strong>Model Evaluation Plots (MEP)</strong></p><p><em>MEPs provide a comprehensive evaluation of model performance by combining multiple diagnostic plots in a single display. They include:</em></p><ul><li><em>scatterplots</em></li><li><em>time series plots</em></li><li><em>histograms</em></li><li><em>other diagnostic plots to assess various aspects of model skill</em></li></ul><p><strong>Reliability diagrams<br></strong><em>The reliability diagram groups the forecasts into bins according to the issued probability (horizontal axis). The frequency with which the event was observed to occur for this sub-group of forecasts is then plotted against the vertical axis.</em></p><p><strong>Sharpness diagrams<br></strong><em>The sharpness diagrams show the relative frequency with which the event has been predicted (over the reference period and at all gridpoints) with different levels of probability.</em></p><p><strong>Continuous Ranked Probability Score (CRPS) diagrams</strong><em>CRPS diagrams provide a visual representation of the accuracy and reliability of probabilistic forecasts. They compare the cumulative distribution function of observed data against the predicted cumulative distribution function, helping to evaluate the quality of probabilistic models.</em></p><p><strong>Hierarchical clustering diagrams<br></strong><em>Hierarchical clustering diagrams group models based on their similarity in representing observed data. They provide insights into the relationships and similarities between different models, facilitating model selection and ensemble construction.</em></p><p><strong><em>Conclusion</em></strong><em><br>By combining machine learning algorithms with search techniques, we unlock the potential to exceed the forecasting capabilities of traditional baseline models.</em></p><p><em>Through dataset preparation, feature engineering, training, evaluation with search algorithms, and iterative fine-tuning, we can discover an algorithm that delivers improved forecasting accuracy.</em></p><p><em>This integration of machine learning and search algorithms empowers decision-makers across industries, enabling them to make more informed choices based on reliable predictions.</em></p><p><em>The marriage of these techniques paves the way for advanced forecasting solutions and opens doors to new possibilities in the field of time series analysis.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bb69a95f490" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probability Theory for Data Science Crash Course]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/probability-theory-in-a-nutshell-crash-course-9d62aa896806?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d62aa896806</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 00:17:48 GMT</pubDate>
            <atom:updated>2023-07-22T20:09:04.925Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>This course covers:</em></strong></p><ol><li><em>The basics of probability</em></li></ol><p><em>2. Combinatorics</em></p><p><em>3. Bayesian Inference</em></p><p><em>4. Discrete Distributions</em></p><p><em>5. Continuous Distributions</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/0*dHIvDONO3u3bwz46.gif" /></figure><p><strong><em>Without further ado let’s dive in…</em></strong></p><h3><em>1. The Basics of Probability</em></h3><blockquote><strong>What is probability all about ?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*Gl-ZCFaHrjN0zW27.gif" /></figure><p><em>Probability is about quantifying the likelihood of an event occurring.</em></p><p><em>We always express it numerically as follows:</em></p><p><strong><em>0 &lt;- not occurring</em></strong><em> 0.1 .… 0.5 ……….0.8…. </em><strong><em>-&gt; 1 occurring</em></strong></p><p><strong><em>Probability(of Event) = favorable outcomes / sample space</em></strong></p><blockquote><strong>Coin Example</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*Mupy65s1_C27Rz9_.gif" /></figure><p><em>If (A) → Coin → Heads / Tails → 2 possible outcomes</em></p><p><em>Probability(of Head) = 1/2 = 0.5</em></p><blockquote><strong>Six Sided Die</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*P7Tap2PBAu5t1WVb.gif" /></figure><p><em>A → 6 sided DIE → 6 possible outcomes</em></p><p><em>P(A) = 1/6 = approx. 0.167</em></p><p><em>P(A) = 2/6 = approx. 0.33</em></p><blockquote><strong>Winning a Lottery</strong></blockquote><p><em>P(A) → Winning the Lottery = amount of tickets we buy divided by the total possible outcomes.</em></p><p><em>Let’s assume there exist 175 million tickets in a given lottery we want to participate in.</em></p><p><em>Assuming we only bought one ticket , we can calculate the probability of winning the lottery as follows:</em></p><p><em>P(Winning Ticket) = 1 / 175 000 000 = approx. 0.000 000 005</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*GbzqUWH6EW-JByBP.gif" /></figure><blockquote>…Good luck!</blockquote><blockquote><strong>Given Two Independent Events occurring at the same time</strong></blockquote><p><em>P(A and B) = P(A) * P(B)</em></p><p><em>they are equal to the product of each individual event</em></p><blockquote><strong>EXPERIMENAL PROBABILITY</strong></blockquote><p><em>P(A) = successful trails / all trials</em></p><blockquote><strong>EXPECTED VALUE</strong></blockquote><p><em>is the average outcome we expect to occur if we run an experiment many times.</em></p><p><em>single trial → (ex.: flip a coin) and record outcome</em></p><p><em>multiple trials → experiment</em></p><blockquote><strong>Categorical outcomes (ex.: drawing a spade from a deck of cards )</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*rMUKtEZjRXFIk8d7.gif" /></figure><p><em>E(A) = P(Drawing Spade) * number of trials</em></p><blockquote><strong>Numerical outcomes (ex.: hitting a 3 layered target with an arrow)</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*AMsyDZl4SlEszGlr.gif" /></figure><p><em>where each layer has points and assigned probability of hitting it</em></p><p><em>A = 10 points ; → P(A) = 0.5</em></p><p><em>B = 20 points; → P(B) = 0.4</em></p><p><em>C = 100 points; → P(C) = 0.1</em></p><blockquote>E(X) = P(A) * A + P(B) * B + P(C) * C</blockquote><p><em>E(X) = 0.5 * 10 + 0.4 * 20 + 0.1 * 100</em></p><p><em>E(X) = 23</em></p><blockquote><strong>Understanding Expected Value</strong></blockquote><p><em>The EV of a random variable gives a measure of the center of the distribution of the variable.</em></p><p><em>Essentially, the EV is the long-term average value of the variable. Because of the law of large numbers, the average value of the variable converges to the EV as the number of repetitions approaches infinity. EV is also known as expectation, the mean or the first moment.</em></p><blockquote><strong>What is Probability Frequency Distribution ?</strong></blockquote><p><em>a collection of the probabilities for each possible outcome of an event.</em></p><p><em>We need the probability frequency distribution to try and predict future events when the expected value is unattainable.</em></p><p><em>Frequency is the number of times a given value or outcome appears in the sample space.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*Vxonm0dhXKKUQZqX.gif" /></figure><blockquote><em>By rolling 2 six sided dice we can construct the following table :</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/0*yKVC-XyKeVyz7x2f" /></figure><p><em>We can notice that number 7 occurs six times in the table.</em></p><p><em>To confirm it we can construct the frequency distribution table as follows:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*boRWEiogN8sTAHVSDbGHuQ.png" /></figure><p><em>The frequency represents the number of times the value features in the sample space.</em></p><blockquote><strong>How do we obtain the probability frequency distribution from the frequency distribution table?</strong></blockquote><p><em>By dividing every frequency by the size of the sample space. Think about the “favoured over all” formula.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*mq6LhhqAciL2suRpNETWeA.png" /></figure><blockquote><strong>What are Complements ?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*j4LpXiaXOQsf5Glj.png" /></figure><p><em>The complement of an event is everything an event is not. It helps to complete the rest of the sample space.</em></p><p><em>We denote the complement of an event with an apostrophe.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*ptYUMSraO3rFIjWJkY3Yvw.png" /></figure><blockquote><strong>Characteristics of complements:</strong></blockquote><ul><li><em>Can never occur simultaneously.</em></li><li><em>Add up to the sample space → </em><strong><em>(A + A’ = Sample space)</em></strong></li><li><em>Their probabilities add up to 1 → </em><strong><em>(P(A) + P(A’) = 1)</em></strong></li></ul><p><strong><em>Examples</em></strong>:</p><ul><li><em>Assume event A represents drawing a spade, so </em><strong><em>P(A) = 0.25.</em></strong></li><li><em>Then, A’ represents not drawing a spade, so drawing a club, a diamond or a heart. </em><strong><em>P(A’) = 1 — P(A), so P(A’) = 0.75.</em></strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/81/0*hKoRprDYfQ-ozak2.jpg" /></figure><p>When the event is <strong>Heads</strong>, the complement is <strong>Tails</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/0*bAdGrSZAZJmgDrbZ.gif" /></figure><p>When the event is <strong>{Monday, Wednesday}</strong> the complement is <strong>{Tuesday, Thursday, Friday, Saturday, Sunday}</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/0*Wl7-QWaKYOvXZO7m.jpg" /></figure><p>When the event is <strong>{Hearts}</strong> the complement is <strong>{Spades, Clubs, Diamonds, Jokers}</strong></p><blockquote><strong>Why is the Complement Useful?</strong></blockquote><p><em>Throw two dice.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/0*19NlK6FoFnHycDYb.gif" /></figure><p><strong><em>What is the probability the two scores are different?</em></strong></p><p><em>Different scores are like getting a </em><strong><em>2 and 3</em></strong><em>, or a </em><strong><em>6 and 1</em></strong><em>. It is a long list:</em></p><p><em>A = { (1,2), (1,3), (1,4), (1,5), (1,6),<br> (2,1), (2,3), (2,4), (2,5), (2,6),<br>(3,1), (3,2), … etc ! }</em></p><p><em>But the complement (which is when the two scores are the same) is only </em><strong><em>6 outcomes</em></strong><em>:</em></p><p><em>A’ = { (1,1), (2,2), (3,3), (4,4), (5,5), (6,6) }</em></p><p><em>And its probability is:</em></p><p><em>P(A’) = 636 = 16</em></p><p><em>Knowing that P(A) and P(A’) together make 1, we can calculate:</em></p><p><em>P(A) = 1 − P(A’) = 1 − 16 = 56</em></p><p><em>So in this case (and many others) it is easier to work out </em><strong><em>P(A’)</em></strong><em> first, then calculate </em><strong><em>P(A) = 1 − P(A’)</em></strong></p><h3>2. Combinatorics</h3><p><em>Combinatorics deals with combinations of objects from a specific finite set.</em></p><p><em>3 integral parts that make up Combinatorics are Permutations , Variations and Combinations.</em></p><p><em>There are certain restrictions that we can apply to form combinations as follows:</em></p><p><em>- repetition</em></p><p><em>- order</em></p><p><em>- some different criterio</em>n</p><blockquote><strong>Brief Example :</strong></blockquote><p><em>When picking items from a collection to form another collection, if the </em><strong><em>order of the elements matters, it is a permutation or variation</em></strong><em>(e.g., choosing a lock password).</em></p><p><strong><em>If the order doesn’t matter, it is a combination</em></strong><em> (e.g., making a fruit salad). The new collection can contain duplicates or repetitions (e.g., using duplicate numbers in a lock password), or it can exclude duplicates altogether (e.g., race winners cannot be duplicated).</em></p><blockquote><strong>Permutations</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*KX_wwn-BJdIBZj1A.gif" /></figure><p><em>The purpose of permutations is to determine the total number of arrangements or orderings possible.</em></p><p><em>Permutations are useful in scenarios where the order of the elements matters, such as creating unique passwords, arranging a sequence of events, or assigning distinct positions to objects.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*Q930In4KJiVW7na1H8v0Ag.png" /></figure><blockquote><strong>Characteristics of Permutations:</strong></blockquote><ul><li><em>Arranging all elements within the sample space.</em></li><li><em>No repetition.</em></li><li><em>𝑃 𝑛 = 𝑛 × 𝑛 − 1 × 𝑛 − 2 × ⋯ × 1 = 𝑛! (Called “n factorial”)</em></li><li><strong><em>Example:</em></strong><em> If we need to arrange 5 people, we would have P(5) = 120 ways of doing so.</em></li></ul><blockquote><strong>Permutations Concept Cheat Sheet</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*1k5TI3tzGOHimEDj.png" /></figure><blockquote><strong>Solving Factorials — Refresher :)</strong></blockquote><p><em>Factorials express the product of all integers from 1 to n and we denote them with the “!” symbol.</em></p><p><em>Negative numbers don’t have a factorial and 0! is always equal to 1.</em></p><p><em>In case you need a more in depth introduction regarding factorials check out the following link:</em></p><p><a href="https://www.mathsisfun.com/numbers/factorial.html">Factorial !</a></p><blockquote><strong>Variations</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/0*MFEjoJH483Yi93BL.gif" /></figure><p><em>Variations, similar to permutations, involve selecting a subset of items from a set and where the order also does matter.</em></p><p><em>The purpose of variations is to determine the number of arrangements with a specific order when selecting a subset.</em></p><p><em>Variations are useful when considering ordered subsets of elements, allowing for flexibility in the selection process.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*BpGPuJQ-g87fslzPz2CH7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*0kebfW_BWyzKKd2a_J-TZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*sOtewfN_r5xJoMxAGfchow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*KRJJ3mpAdDtTWCKPjg_GYw.png" /></figure><blockquote><strong>Combinations</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*ZkaHIjfqL-lx8tey.gif" /></figure><p><em>Combinations are used to calculate the total number of ways to select subsets from a larger set, without considering the order of the elements.</em></p><p><em>The purpose of combinations is to determine the number of possibilities when the order doesn’t matter.</em></p><p><em>Combinations are often used in situations such as creating teams, choosing a committee, or selecting items for a group where the arrangement is irrelevant</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*DezCMGkeynuKF6XXi5PD2g.png" /></figure><blockquote><strong>Combinations with Repetition</strong></blockquote><p><em>Keep in mind that in this case the repetition is allowed , however we do not care abut the order of the distinct values in the sample space.</em></p><p><em>We use combinations with repetition when the events we are dealing with, have sufficient quantity of each of the distinct values in the sample space.</em></p><p><em>In special cases we can have repetition in combinations and for those we use a different formula than previously.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*mmDYGRrAIWj5X3XgbEup3Q.png" /></figure><p><em>Now that you know what the formula looks like, I am going to walk you through the process of deriving this formula from the Combinations </em><strong><em>without repetition</em></strong><em> formula.</em></p><p><em>This way you will be able to fully understand the intuition behind and not have to bother memorizing it.</em></p><blockquote><strong>Combinations without Repetition</strong></blockquote><blockquote><em>Formula : C(n, p) = n! / (p! * (n — p)!)</em></blockquote><blockquote><strong>Combinations Concept Cheat Sheet</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*e1iLeJX3Ptei1ByM.png" /></figure><blockquote><strong>Symmetry of Combinations</strong></blockquote><p><em>Picking p many elements out of n, is the same as omitting n- p many elements.</em></p><h4>__________________________________________________________</h4><p><em>Before we dive in into Bayesian Inference as a part of this crash course I strongly encourage you to explore the Frequentist Inference approach on your own.</em></p><p><em>These 4 are great resources:</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXmHPcdR3bWc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXmHPcdR3bWc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXmHPcdR3bWc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b02eb6c755199325ce8d921f3c074666/href">https://medium.com/media/b02eb6c755199325ce8d921f3c074666/href</a></iframe><ul><li><a href="https://leanpub.com/modeling-mindsets">Modeling Mindsets</a></li><li><a href="https://en.wikipedia.org/wiki/Frequentist_inference">Frequentist inference - Wikipedia</a></li><li><a href="https://en.wikipedia.org/wiki/Frequentist_probability">Frequentist probability - Wikipedia</a></li></ul><p><strong>but.. just to scratch the surface</strong></p><p><em>Frequentist inference is a statistical approach that focuses on analyzing data based on repeated sampling.</em></p><p><em>It emphasizes the long-run frequency of events to make inferences about population(“all” the data available) parameters.</em></p><p><em>In frequentist inference, statistical hypotheses are formulated and tested using methods such as hypothesis testing and confidence intervals.</em></p><p><em>Notations commonly used in frequentist inference include</em></p><ul><li><em>X̄ for sample mean</em></li><li><em>μ for population mean</em></li><li><em>s for sample standard deviation</em></li><li><em>σ for population standard deviation</em></li><li><em>p for population proportion</em></li></ul><p><em>Frequentist inference aims to provide objective and reliable statistical inference based on observable data and probability theory.</em></p><p><strong><em>and last but not least 2 bonuses:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*JUigiJwZbN9zo-NIIeIyvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/1*uY2aN25qBEIww9eyBiFV2A.png" /></figure><h3>3. Bayesian Inference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*9oIgSPSeOHx92Ov8.gif" /></figure><blockquote><strong>Bayesian Notation</strong></blockquote><p><em>A set is a collection of elements, which hold certain values. Additionally, every event has a set of outcomes that satisfy it.</em></p><p><em>The null-set (or empty set), denoted ∅, is an set which contain no values.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*PHrrhltwWJzBU_hKtZSN5A.png" /></figure><blockquote><strong>Multiple Events</strong></blockquote><p><em>The sets of outcomes that satisfy two events A and B can interact in one of the following 3 ways.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d2U1cfbUe6qeKk171eJb4g.png" /></figure><ol><li><em>If the two circles never touch , the two events can never happen simultaneously for ex.: Event A occuring guarantees that event B is not occuring and vice versa.</em></li></ol><p><em>2. If they intersect , the two events can occur at the same time.</em></p><p><em>3. If they overlap means one event can only occur if the other does as well.</em></p><blockquote><strong>Intersection</strong></blockquote><p><em>The</em><strong><em> intersection</em></strong><em> of two or more events expresses the set of outcomes that satisfy all the events simultaneously. Graphically, this is the area where the sets intersect.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*awK2w9myvZqsDvsI4ywCzw.png" /></figure><blockquote><strong>Union</strong></blockquote><p><em>The </em><strong><em>union </em></strong><em>of two or more events expresses the set of outcomes that satisfy at least one of the events. Graphically, this is the area that includes both sets.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qDnPd90O_OdxEZBTdua1Pg.png" /></figure><blockquote><strong>Mutually Exclusive Sets</strong></blockquote><p><em>Sets with no overlapping elements are called mutually exclusive. Graphically, their circles never touch.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0nU0xJV_c26o6BAEFKMOw.png" /></figure><blockquote><strong>Independent and Dependent Events</strong></blockquote><p><strong><em>Independent : </em></strong><em>The theoretical probability remains unaffected by other events</em></p><p><strong><em>Dependent :</em></strong><em> Probabilities of dependent events vary as conditions change</em></p><p><em>If the likelihood of event A occurring (P(A)) is affected event B occurring, then we say that A and B are </em><strong><em>dependent events.</em></strong></p><p><em>Alternatively, if it isn’t — the two events are </em><strong><em>independent</em></strong><em>.</em></p><p><em>We express the probability of event A occurring, given event B has occurred the following way 𝑷 (𝑨 /𝑩) . We call this the conditional probability.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*sRUyRQ_b6qEk7yDwfQEvHw.png" /></figure><h4>The probability of an event changes depending on the information we have!</h4><blockquote><strong>Conditional Probability</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IYkywMygZcO084oMCkN8ew.png" /></figure><blockquote><strong>Law of total probability</strong></blockquote><p><em>The law of total probability dictates that for any set A, which is a union of many mutually exclusive sets 𝐵1,𝐵2, … , 𝐵𝑛, its probability equals the following sum.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*V_kOLYNIpylWMs8vBu5M7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*MlKLqY7eqFbw5SvQLdvvpg.png" /></figure><blockquote><strong>Additive Law</strong></blockquote><p><em>The additive law calculates the </em><strong><em>probability of the union</em></strong><em> based on the probability of the individual sets it accounts for.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*pQvTrw90gKb3qd6heKbhfA.png" /></figure><blockquote><strong>The Multiplication Rule</strong></blockquote><p><em>The multiplication rule calculates the </em><strong><em>probability of the intersection</em></strong><em> based on the conditional probability.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*xnOultN91U7eP3a1-HXQpQ.png" /></figure><blockquote><strong><em>Bayes’ Law</em></strong></blockquote><p><em>Bayes’ Law helps us understand </em><strong><em>the relationship between two events by computing the different conditional probabilities.</em></strong><em> We also call it </em><strong><em>Bayes’ Rule or Bayes’ Theorem.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kM4P1rhwR7gakiIxo0zZgg.png" /></figure><blockquote><strong>Real Life Example — Bayesian Inference</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*ngcVXOSdYw-0pyiPcIPb9g.png" /></figure><ol><li><strong><em>What is the likelihood for a male student to be accepted?</em></strong></li></ol><p><em>P(A) = 634 / 2590</em></p><p><em>P(A) = 0.24</em></p><p><strong><em>2) What is the likelihood for a female student to be accepted?</em></strong></p><p><em>P(B) = 741 / 3088</em></p><p><em>P(B) = 0.23</em></p><p><strong><em>3) Does gender have an effect on your chances of being accepted?</em></strong></p><p><em>Slightly but is relatively even as :</em></p><p><em>P(A) = approx. 24 % for male <br>P(B) = approx. 23 % for female</em></p><p><strong><em>4) Are first time freshmen men or women more likely to enroll?</em></strong></p><p><em>P(A) = 217 / 634 = 34.23 %</em></p><p><em>P(B) = 263 / 741 = 35.49 %</em></p><p><em>The higher enrollment rate makes sense, given the lower acceptance rate among female applicants.</em></p><blockquote><strong>Onto table C2</strong></blockquote><p><strong><em>1)What is the likelihood of being offered a place on the waitlist?</em></strong></p><p><em>P(A) = 1299 / ( 3088 + 2590 ) = 0.2287 = 28.87 %</em></p><p><strong><em>2) What is the likelihood of getting admitted, having accepted a place on the waitlist?</em></strong></p><p>P(B) = 33 / 629 = 0.0524 = 5.24 %</p><p><strong><em>3) What is the likelihood of getting admitted, having been offered a place on the waitlist?</em></strong></p><p>P(C) = 33 / 1299 = 0.0254 = 2.54 %</p><h3><strong>Probability Distributions</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*gsRj28FKC9L1OA9e.gif" /></figure><blockquote><em>A distribution shows the possible values a random variable can take and how frequently they occur.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*CtllNBp003j5gIgCZbvXCw.png" /></figure><blockquote><strong>Probability Frequency Distribution</strong></blockquote><p><em>Just to recall from the very first chapter:</em></p><p><em>Probabilities measure the likelihood of an outcome depending on how often it is featured in a sample space</em></p><p><strong><em>We define distributions using two characteristics :</em></strong></p><p><em>mean → average value</em></p><p><em>variance → how spread out the data is</em></p><blockquote><strong>Population vs Sample</strong></blockquote><p><em>Population data → “all” the data</em></p><p><em>Sample data → part of population data</em></p><blockquote><strong>Moments you need to know! ( Characteristics of a distribution)</strong></blockquote><p>Moments describe how the probability mass of a random variable is distributed.</p><p>0. The zeroth moment, total mass, quantifies the fact that all distribution’s have a total mass of one.</p><ol><li><em>The first moment, the mean, specifies the distribution’s location, shifting the center of mass left or right.</em></li><li><em>The second moment, variance, specifies the scale or spread; loosely speaking, flatter or more spread out distributions are “more random”.</em></li><li><em>The third moment, skewness, quantifies the relative size of the two tails of a distribution; the sign indicates which tail is bigger and the magnitude indicates by how much.</em></li><li><em>The fourth moment, kurtosis, captures the absolute size of the two tails. Higher standardized moments simply recapitulate the information in skewness and kurtosis; by convention, we ignore these in favor of the third and fourth standardized moments.</em></li></ol><p><em>Finally, moments are important theoretically because they provide an alternative </em><strong><em>way to fully and uniquely specify a probability distribution</em></strong><em>, a fact that is intuitive if you </em><strong><em>understand how moment’s quantify a distribution’s location, spread, and shape.</em></strong></p><p><a href="https://gregorygundersen.com/blog/2020/04/11/moments/">Understanding Moments</a></p><blockquote><strong>Types of Distributions</strong></blockquote><p><em>Certain distributions share characteristics, so we separate them into types. The well-defined types of distributions we often deal with have elegant statistics. We distinguish between two big types of distributions based on the type of the possible values for the variable — discrete and continuous.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SgTQqSOozPwuMv9V_os8xA.png" /></figure><p><strong><em>Let’s review few examples of discrete distributions to give you a better understanding.</em></strong></p><h3><strong><em>4. Discrete Distributions</em></strong></h3><p><em>Discrete Distributions have finitely many different possible outcomes. They possess several key characteristics which separate them from continuous ones.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qBAN6vULSJWGM_kNHDlRRg.png" /></figure><blockquote><strong>Notation:</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*_v4FCadDxVaAokVoP9Iz-Q.png" /></figure><blockquote><strong>Uniform Distribution</strong></blockquote><p><em>A distribution where all the outcomes are equally likely is called a Uniform Distribution.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*17nprlNWjioM32TerU1vpA.png" /></figure><blockquote><strong>Bernoulli Distribution</strong></blockquote><p><em>A distribution consisting of a </em><strong><em>single trial and only two possible outcomes</em></strong><em> — success or failure is called a Bernoulli Distribution.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKiq7mGq9amBEIECac-F_A.png" /></figure><blockquote><strong>Binomial Distribution</strong></blockquote><p><strong><em>A sequence of identical Bernoulli events</em></strong><em> is called Binomial and follows a Binomial Distribution</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6tBYzgquEIzqJn0rva7yQ.png" /></figure><p><strong><em>Practical Example</em></strong></p><p><em>Imagine you bought a single stock of general motors and you know from historical data that there exists 60 % probability of stock going up on any given day and 40 % probability of stock going down. By the price going up I mean that the daily closing price is higher than the opening price.</em></p><p><em>With Probability Distribution Function of Binomial Distribution we can calculate the likelihood of stock increasing exactly3 times in 5 trading bussines days.</em></p><p>y = 3<br>n = 5 <br>p = 0.6</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*UIN9Uc5B2L3AFvuDFhXHdQ.png" /></figure><p><em>By plugging in the values into the formula we get </em><strong><em>34.56 % chance</em></strong><em> of getting exactly 3 increases.</em></p><p><em>Calculating Expected Value for Binomial Event.</em><br>E(Y) = (n * p)<br>E(Y) = ( 5 * 0.6)<br>E(Y) = 3</p><p><em>Calculating variance and standard deviation.</em></p><p>Var(Y) = n * p * (1 — p)<br>Var(Y) = 5 * 0.6 * (1 – 0.6) = 1.2</p><p><em>Knowing the expected value and variance we can make more accurate future forecasts.</em></p><blockquote><strong>Poisson Distribution</strong></blockquote><p><em>When we want to know </em><strong><em>the likelihood of a certain event occurring over a given interval of time or distance</em></strong><em>, we use a Poisson Distribution.</em></p><p><em>The graph of the Poisson Distribution start from 0 on the X-axis , because we are measuring the frequency of occurrence over a given interval of time or distance which cannot be negative.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-isZeM_Rx3bL3eTfb_f3yw.png" /></figure><p><strong><em>Practical Example</em></strong></p><p><em>Q &amp; A Example during class.<br>Students questions per day : usually 4 <br></em><strong><em>Yesterday : 7 !!!!!</em></strong><em><br>By the sudden spike in interest you think how likely it was to get exactly 7 questions.</em></p><p><em>lambda λ = 4<br>Probability Function for Poisson Dist. = 4<br>Euler’s Number / Napier’s constant ( fixed value ) = always equals 2.72<br>Looking for Probability of getting 7 Questions = y(7)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*YYSOMT6mQEPZrrjKlu0W_g.png" /><figcaption>Probability Function Formula for Poisson Distribution</figcaption></figure><p><strong><em>P(7) = ( (4**7) * (2.72**(-4)) ) / 7! = 0.059 = approx. → 0.06 or 6 % chance of receiving exactly 7 questions.</em></strong></p><blockquote><strong>Note!</strong></blockquote><p><strong><em>Both the mean and variance for a Poisson Distribution are equal to lambda (𝜆).</em></strong></p><p><em>More about it in here:</em><br><a href="https://www.scribbr.com/statistics/poisson-distribution/">https://www.scribbr.com/statistics/poisson-distribution/</a></p><h3><strong>5. Continuous Distributions</strong></h3><p><em>If the possible values a random variable can take are a </em><strong><em>sequence of infinitely many consecutive values</em></strong><em>, we are dealing with a </em><strong><em>continuous distribution.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5vHYf7uXpWLDDSMx7GnBmQ.png" /></figure><p><em>Up until now I referred to Probability Density Function as Probability Function. Now it is time to introduce the</em><strong><em> Cumulative Distribution Function.</em></strong></p><blockquote><strong>Cumulative Distribution Function</strong></blockquote><p><em>Since it is called “cumulative” this function encompasses everything up to a certain value. It represents probability of the random variable being lower than or equal to a specific value. CDFs are useful especially , if we want to estimate the probability of some interval.</em></p><p><em>In short , the CDF represents the area under the curve of the PDF.</em></p><blockquote><strong>Integrals</strong></blockquote><p><em>Graphically the area under the density curve would represent the chance of getting a value within that interval. We find this area by computing the integral of the density curve over the interval from “a” to “b” as on the image below.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*FJlutlNJJ631yiXAplJAFA.png" /></figure><p><a href="https://en.wikipedia.org/wiki/Integral">Integral - Wikipedia</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FrfG8ce4nNh0%3Fstart%3D162%26feature%3Doembed%26start%3D162&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DrfG8ce4nNh0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FrfG8ce4nNh0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/22cd9820baa9b8f549621206f5484617/href">https://medium.com/media/22cd9820baa9b8f549621206f5484617/href</a></iframe><blockquote><strong>Normal Distribution</strong></blockquote><p>A Normal Distribution represents a distribution that most natural events follow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zovon58CGjnh8_ZzCJuxaQ.png" /></figure><blockquote><strong>Standardizing a Normal Distribution</strong></blockquote><p><em>To standardize any normal distribution we need to transform it so that </em><strong><em>the mean is 0</em></strong><em> and the </em><strong><em>variance and standard deviation are 1.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*J3ABRWR8fPO81lzU2gCQBw.png" /></figure><blockquote><strong>Students’ T Distribution</strong></blockquote><p><em>A small sample size approximation of a Normal Distribution. Meaning where there is no sufficient data available about a variable, which historically follows a Normal Distribution. It accommodates the occurrence of values far away from the mean (we call it “fatter tails”). It is applicable when conducting statistical analysis , hypothesis testing with limited data and CDF table (T-table) construction.</em></p><p><strong><em>k</em></strong> <em>stands for degrees of freedom</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mTynzBnyF5LCiMtEnKh5bw.png" /></figure><blockquote><strong>Chi-Squared Distribution</strong></blockquote><p><em>A Chi-Squared distribution is an asymmetric type of distribution. This distribution is applicable in hypothesis testing and computing confidence intervals or for goodness of fit of categorical values. Finding the square root of Chi-Squared Distribution gives us the Student’s T Distribution.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Vct0hPTGPEacOEDBP9Nyw.png" /></figure><blockquote><strong>Exponential Distribution</strong></blockquote><p><em>The Exponential Distribution is usually observed in events which significantly change early on.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bsTZiKiEMtwo3IFM4MHbWA.png" /></figure><blockquote><strong>Logistic Distribution</strong></blockquote><p><em>The Continuous Logistic Distribution is observed when trying to determine how continuous variable inputs can affect the probability of a binary outcome.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mU_zBN3VlAr6vvibNT4DMw.png" /></figure><p><strong><em>CDF of a Logistic Distribution</em></strong></p><ul><li><em>It follows an S-shape and plateaus around the 1 value.</em></li><li><em>The probability drastically starts to increase once we reach values close to the mean.</em></li><li><em>The steeper the curve is, the faster it reaches value close to absolute certainty (1).</em></li></ul><p><strong><em>Practical Example</em></strong></p><p><em>Analyze if the average speed in MPs of a tennis player’s serve play a crucial role in the outcome of the match.</em></p><p><strong><em>Expectation</em></strong><em> dictates that sending the ball with higher velocity leaves opponents with a shorter period to respond. This usually results in a better hit , which could lead to a point for the server.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*-sDBL0wzEJFrvuJiLO0cag.png" /></figure><p><strong>Reality </strong><em>dictates that in order to reach the highest speeds , tennis player often give up some control over the shot. So are less accurate.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T2a6WwmIiuuvyX3JMctUww.png" /></figure><p><strong><em>Theory </em></strong><em>Suggests the exist some “Optimal Speed” that makes the serve still accurate. So most of the shots we convert into points , will have similar velocities.</em></p><p><em>As tennis players go further away from the optimal speed their shot become too slow and easy to handle or inaccurate.</em></p><p><em>Parameters of logistic distributions are:</em></p><p>µ — mean<br>s — scale parameter</p><p><em>In this case the mean would represent the optimal speed and the scale parameter would dictate how gentle we can be with the hit. You can plot it , define the values and examine the results on your own.</em></p><blockquote><strong>Conclusion</strong></blockquote><p><em>What I have done so far is to lay the foundations of probability theory as well as introduce the most frequently used distributions in the field of data science.</em></p><p><em>Having mastered that, you should be ready to dive deeper and conduct independent research on the more difficult sections of probability and statistics and/or explore different probability distributions.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d62aa896806" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Functional data analysis]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/functional-data-analysis-f480e09c9bc8?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/f480e09c9bc8</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Tue, 26 Apr 2022 10:51:43 GMT</pubDate>
            <atom:updated>2022-04-26T22:45:18.621Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Functional data analysis</strong> (FDA) is a branch of <a href="https://en.wikipedia.org/wiki/Statistics">statistics</a> that analyzes data providing information about curves, surfaces or anything else varying over a continuum. In its most general form, under an FDA framework, each sample element of functional data is considered to be a random function. The physical continuum over which these functions are defined is often time, but may also be spatial location, wavelength, probability, etc. Intrinsically, functional data are infinite dimensional. The high intrinsic dimensionality of these data brings challenges for theory as well as computation, where these challenges vary with how the functional data were sampled. However, the high or infinite dimensional structure of the data is a rich source of information and there are many interesting challenges for research and data analysis.</p><p>WHAT IS FUNCTIONAL DATA ?</p><p><em>In (</em><a href="http://faculty.bscb.cornell.edu/~hooker/ShortCourseHandout.pdf"><em>Hooker, 2017</em></a><em>) it is defined, that “Functional data is multivariate data with an ordering on the dimensions”. It means that this type of data consists of curves varying over a continuum, such as time, frequency, or wavelength. Thus, functional data is often present when measurements at various time points are analyzed. The curves of functional data are usually interdependent, which means that the measurement at a point, e.g., 𝑡𝑖+1ti+1, depends on measurements at some other points, e.g., 𝑡1,…,𝑡𝑖;𝑖∈ℕt1,…,ti;i∈N.</em></p><p><em>As the most well-known machine learning techniques generally do not emphasize the interdependence between features, they are often not well suited for such tasks. Application of these techniques can lead to poor performance. Functional data analysis, on the other hand, tries to address this issue either by using algorithms particularly tailored to functional data or by transforming the functional covariates into a continuum (e.g., time, wavelength) nondependent feature space. For a more in-depth introduction to functional data analysis see, e.g. </em><a href="http://rd.springer.com/article/10.1007/BF02293704"><em>When the data are functions</em></a><em> (Ramsay, J.O., 1982).</em></p><p><a href="https://mlr.mlr-org.com/articles/tutorial/functional_data.html#:~:text=In%20(Hooker%2C%202017)%20it,time%2C%20frequency%2C%20or%20wavelength.">Functional Data</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f480e09c9bc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gabor Transform]]></title>
            <link>https://medium.com/@yoshiyoshiyoshi/gabor-transform-a3ebdc4324d9?source=rss-1c9ce7556478------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3ebdc4324d9</guid>
            <dc:creator><![CDATA[Data Science Pro]]></dc:creator>
            <pubDate>Wed, 13 Apr 2022 20:24:28 GMT</pubDate>
            <atom:updated>2022-04-13T20:24:28.436Z</atom:updated>
            <content:encoded><![CDATA[<p>………</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEfWnEldTyPA%3Fstart%3D648%26feature%3Doembed%26start%3D648&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEfWnEldTyPA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEfWnEldTyPA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6d593197c6885c7d588412c330dc05b/href">https://medium.com/media/e6d593197c6885c7d588412c330dc05b/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3ebdc4324d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>