<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yasser Mohammad on Medium]]></title>
        <description><![CDATA[Stories by Yasser Mohammad on Medium]]></description>
        <link>https://medium.com/@yasserfarouk?source=rss-511819d0237d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*k8QlZ6gMMcg_-HZl.</url>
            <title>Stories by Yasser Mohammad on Medium</title>
            <link>https://medium.com/@yasserfarouk?source=rss-511819d0237d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:36:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yasserfarouk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reinforcement Learning and Automated Negotiation — A match made in gymnasium]]></title>
            <link>https://yasserfarouk.medium.com/reinforcement-learning-and-automated-negotiation-a-match-made-in-gymnasium-076a205cd1c0?source=rss-511819d0237d------2</link>
            <guid isPermaLink="false">https://medium.com/p/076a205cd1c0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[competitions-for-student]]></category>
            <category><![CDATA[reinforecment-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[supply-chain-management]]></category>
            <dc:creator><![CDATA[Yasser Mohammad]]></dc:creator>
            <pubDate>Thu, 15 Feb 2024 08:55:29 GMT</pubDate>
            <atom:updated>2024-02-15T08:57:17.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reinforcement Learning and Automated Negotiation — A match made in gymnasium</h3><p>Reinforcement learning is probably one of the most successful ideas in the history of computing. It allowed us lowly computer geeks to build agents that beat or at least rival world champions in chess, go, backgammon, poker, Atari games, and, coolest of all, <a href="https://github.com/google-deepmind/pysc2">starcraft</a>. It has its role in the rise of generative AIs like ChatGPT through RLHL (Reinforcement Learning from Human Feedback). Nevertheless, RL is still far from achieving its full potential in the real world. One reason is that the world is messy, much messier than these games with well-defined rules. In this post, I will introduce you to a new game for RL that is directly applicable to the real world. Find a solution today, apply it tomorrow. This new game is technically challenging, just at the edge of the current state of the art in RL and multiagent RL research. If you are looking for a new challenge that is more than dogs vs. cats, look no further.</p><h4>Why do we need another RL environment</h4><p>This section goes through the history of RL in games to conclude that we need a new type of environment that is half-way between a Poker game and Atari.</p><h4>Initial successes of RL in Games</h4><p>Reinforcement Learning is one of the most successful machine learning techniques for strategic reasoning and game playing problems. It was successfully applied to many games including Chess, Shogi, Diplomacy, and, recently, Sratego. RL shows promise in physical games as well including table tennis, and hockey. Most of these systems were designed to handle <strong>a single game</strong>. The lessons learned from mastering each of these games did help in mastering the next but usually with new insights added to handle the specific strategic features of the new game.</p><h4>Generalizing to Multiple Games</h4><p>Some methods proved successful in multiple games. For example AlphaZero managed to beat specialist agents in Chess, Go, and Shogi. MuZero was able to master Chess, Go, Shogi and 57 different Atari Games . EfficientZero managed to master 26 different Atari games. These, and similar systems, are a step toward generality because the same learning algorithm is capable of handling multiple games. Nevertheless, an independent training is needed for each game.</p><h4>Generalist Agents</h4><p>Achieving a general game-playing agent is one of the long-term goals of AI research. Recently, more attention was given to RL approaches that allow an agent to play multiple games with a <em>single training</em> on these (or similar) games. For example, Imbala was shown to be able to play multiple Atari games with a single policy.</p><p>One step in this direction was General Game Playing (GGP) methods that present the agent with a formal description of a game in a domain-specific language. The agent is supposed to play competently any specifiable game. RL-GGP allowed testing of different RL algorithms in GGP settings. Monte Carlo Q-Learning was shown to achieve high performance on three GGP games (tic-tac-toe, connect-4 and hex). Generalized ALphaZero used RL to play four games (connect-4, breakthrough, babel, and pacman) specified using GGP and was shown to outperform the state of the art method at the time. Research in RL for GGP is usually focused on small games.</p><p>Recently, Investigated different approaches to train a single model to play 41 different Atari games including decision-transformer based sequence modeling, online RL, offline temporal difference, contrastive representation and behavior cloning . They found that decision transformer methods achieved the best performance and scaling but this required training on both expert and non-expert trajectories. But what can you do when you have neither expert nor non-expert trajectories? Enters <strong>Generalist Environments</strong>.</p><h4>Generalist Environments</h4><p>Training specialist agents in a single environment that provides the same challenge every time (e.g. chess, GO) and training a generalist agent for completely unrelated environments (e.g. Atari) are two extremes for which this post proposes a middle ground.</p><blockquote>We propose a single environment that encapsulates a large set of <em>related</em> problems by changing the <em>configuration</em> of the environment (i.e. a <em>generalist</em> environment). These problems can be considered individually in independent games or collectively as a single game.</blockquote><p>Training a specialist agent for each configuration of a generalist environment is possible but unscalable. Training an agent for all possible configurations of the generalist environment resembles training a generalist agent for multiple games. This type of environment is suitable for training <em>local experts</em> and then learning to activated them as needed for different configurations of the environment which is less explored in RL research. SCML belongs to this class of environments.</p><h4>The SCML competition</h4><p>Negotiations and auctions are the most widely used methods for reaching agreements in most real-world businesses. The simplest form of negotiation is bargaining in which partners exchange offers until one is accepted by everyone becoming a binding agreement (a contract). With With the wide-spread adoption of AI by businesses, internal operations of business units are increasingly being automated and automated negotiation between agents representing these business units is a promising direction that is starting to gain the interest of academics and industrialists.</p><p>Automated negotiations promise faster more efficient agreements and more win-win deals to everyone. The Automated Negotiation Agents Competition (ANAC) started in 2010 in conjunction with the eights International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2010) with the aim of advancing the state of the art in automated negotiation research. The competition succeeded in generating novel research in agent design and expanded to encompass several leagues focusing on different subproblems of automated negotiation.</p><p>Since 2019, a new league was added to ANAC called the <a href="https://scml.cs/brown.edu">Supply Chain Management League: SCML</a>. This competition is based on the real-world problem of negotiating agreements in real markets with applications in transportation, industrial manufacturing, energy production and management, and almost anywhere else buyers and sellers exist.</p><p>Competitors in the SCM league are required to build an agent that controls a factory embedded in a simulated market. Besides controlling production, the agent should negotiate with other agents representing suppliers and consumers to secure supplies and sales. The score of a competitor is the median profit accumulated by all instantiations of her agent in thousands of simulations with varying configurations.</p><figure><img alt="SCML simulation environment showing the production graph" src="https://cdn-images-1.medium.com/proxy/1*JxNQ7GA7hZMz5bi-yQqVmg.jpeg" /><figcaption>SCML simulation environment showing the production graph</figcaption></figure><p>Each instantiation of the SCML simulation defines a multilateral general-sum incomplete-information stochastic game that is determined by the static information (configuration). Agents can be designed/trained to work in a specific configuration (specialist agents), across classes of configurations with common features (e.g. simulations in which the agent is a supplier or simulations with specific number of factories in each production level) leading to <em>local experts</em>, or for <em>any</em> SCML instantiation (generalist agents). That is what makes SCML a <em>generalist environment</em></p><blockquote>The main challenge faced by agents in SCML is a repeated concurrent many-to-many negotiation problem. The agent is negotiating concurrently with all its suppliers and consumers trying to maximize its profit in the current step while keeping an eye in maximizing future profits that can only be generating by future negotiations with the same partners.</blockquote><p>You can find a full description of the game <a href="https://yasserfarouk.github.io/files/scml/y2024/scml2024oneshot.pdf">here</a>.</p><h4>How to participate</h4><p>To participate on this competition, you just need to install a single package (well may be two):</p><pre>pip install scml</pre><p>A nice companion is a streamlit visualizer that can help you debug your code:</p><pre>pip install scml-vis</pre><p>You can then just download the template for a <a href="https://yasserfarouk.github.io/files/scml/y2024/oneshot.zip">classic agent</a> or an <a href="https://yasserfarouk.github.io/files/scml/y2024/oneshot_rl.zip">RL agent</a> and start hacking away.</p><h4>RL for SCML</h4><p>We developed a gymnasium environment that exposes RL as a standard Gymnasium Environment. This SCML environment can be used for both RL and MARL. For simplicity of exposition, we only consider single-agent RL in this blog-post in which the RL agent is allowed to control a single factory (any factory) in each simulation. Extension to multi-agent RL is straightforward.</p><p>SCML exemplifies some of the main challenges of real-world applications of RL:</p><ul><li><strong>High dimensional state space:</strong> The amount of information available to the agent about the environment and other agents is huge.</li><li><strong>System constraints that can never be violated:</strong> The agent should never allow itself to go bankrupt.</li><li><strong>Stochastic systems with partial observability:</strong> The SCML simulation is always stochastic because exogenous contracts are sampled randomly (albeit from a known distribution) every period. Moreover, the agent cannot observe any private information of other agents in the environment.</li><li><strong>Poorly specified reward functions:</strong> The profit calculation method described in Section X defines the change in the agent’s balance in a given period. This is not the <em>utility</em> of the agent because it does not take into account the effect of its carried inventory and the change in future partner behavior based on the agent’s action during this period.</li></ul><p>Moreover, SCML also exhibits the problem of state-space ambiguity discussed in the introduction that plagues many real-world deployments of RL in business applications. Given the richness of information available to the agent, the designer can decide whether or not to use each individual piece of information as a part of the agent’s observation. Moreover, some of the static information (configuration) may be used to construct different <em>training contexts</em> to train multiple RL agents for specific situations, a commonly used strategy in real-world deployments of RL.</p><figure><img alt="Training and using an RL agent for the SCML environment." src="https://cdn-images-1.medium.com/proxy/1*EEjwKk0aJvQ86mnNmsB6pA.jpeg" /><figcaption>Training and using an RL agent for the SCML environment.</figcaption></figure><ul><li><strong>The Environment:</strong> The SCML environment presents an SCML simulation to the agent as a standard RL problem. (Footnote: SCML is available as a standard Farama gymnasium environment <a href="https://gym.openai.com/">https://gym.openai.com/</a>, making it compatible with most RL and MARL environments including SB3, RLLib, tf-agents, and torch-RL.)</li><li><strong>The Agent:</strong> The RL agent controls a single SCML factory receiving observations from the environment and returning actions as counter-offers that are passed to the agent’s suppliers and consumers inhabiting the simulation within the environment.</li><li><strong>Observation Manager:</strong> Selects information to be <em>observed</em> by the agent and encodes it in a suitable RL format (i.e., discrete or continuous numeric values).</li><li><strong>Reward Function:</strong> The only signal directly provided to the agent is profit calculated per period. This signal is sparse Additionally, it doesn’t reflect the value of inventory or effects on future negotiations. The designer can use this component to provide a more informative reward to the agent (i.e., reward shaping).</li><li><strong>Action Manager:</strong> Responsible for <em>decoding</em> agent-generated actions into valid counter-offers for the simulation. Note: We provide a universal action manager usable with an SCML environment creating a continuous or discrete action space as needed.</li></ul><h4>Contextual Training and Deployment</h4><p>As discussed earlier, each SCML configuration defines a unique simulation environment with some shared characteristic with all other simulations (e.g. buyers always prefer lower prices). For example, a <em>L</em>0-agent in a two-levels production graph with 10 competitors and 2 consumers is facing a very different market compared with another <em>L</em>0-agent with 2 consumers and 10 competitors.</p><p>It is unlikely that a single policy or trained model can achieve adequate performance in <em>all</em> possible SCML configurations. Nevertheless, it is impractical to train a dedicated policy for each possible configuration.</p><p>We define a <em>Training Context</em> as a data-structure the defines a set of configurations, an observation manager, an action manager and, optionally, a reward function. In the RL implementation of SCML, environments receive their configuration for simulation generation from a context object. Policies are trained for a specific context (i.e. using environments that share the same context). When deployed, the agent receives a set of contexts and associated trained policies (models). It checks the actual information of the simulation it finds itself inhabiting and selects the most similar context activating its corresponding policy, observation manager and action manager. This process is shown in the following diagram:</p><figure><img alt="Training and using an RL agent for the SCML environment." src="https://cdn-images-1.medium.com/proxy/1*HJ-M86TpZqI1T0LTsUxVhQ.jpeg" /><figcaption>Training and using an RL agent for the SCML environment.</figcaption></figure><p>The process of creating an RL solution for SCML involves the following steps:</p><ol><li><strong>Define Contexts:</strong> Establish training contexts that capture the various market conditions and configurations your agent might encounter within the SCML simulation.</li><li><strong>Design Adapters (Per Context):</strong> For each context, specify:</li></ol><ul><li><strong>Observation Manager:</strong> Function to select and encode relevant information for the agent’s state representation.</li><li><strong>Action Manager:</strong> Function to translate the agent’s actions into valid counter-offers within the simulation.</li><li><strong>Reward Function:</strong> Method to calculate informative rewards that guide the agent’s learning (consider going beyond immediate profit).</li></ul><ol><li><strong>Train Policies (Per Context):</strong> Use your preferred RL algorithm to train a policy for each defined context.</li><li><strong>Deployment and Context Switching:</strong></li></ol><ul><li>Equip the agent with the trained policies and their associated contexts.</li><li>During deployment, have the agent analyze the current simulation to select the most appropriate context, activating the corresponding policy and adapters.</li></ul><p><strong>Illustrative Pseudocode:</strong></p><pre>def create_rl_solution(scml_env):<br>    contexts = define_contexts()  # Function to create list of Training Contexts<br><br>    for context in contexts:<br>        observation_manager = context.observation_manager<br>        action_manager = context.action_manager<br>        reward_function = context.reward_function<br><br>        policy = train_policy(scml_env, observation_manager, action_manager, reward_function)<br><br>    return contexts, policies<br><br>def deploy_agent(agent, scml_env, contexts, policies):<br>    while True:  # Main simulation loop<br>        current_sim_config = scml_env.get_configuration()<br>        best_context = find_matching_context(current_sim_config, contexts)<br><br>        observation = best_context.observation_manager(scml_env)<br>        action = best_context.policy(observation)<br>        counter_offer = best_context.action_manager(action)<br><br>        scml_env.step(counter_offer)<br>        ...</pre><h4>Take-away message</h4><ul><li>SCML provides a new and important type of environment for developing real-world relevant RL and MARL solutions.</li><li>The <a href="https://scml.cs.brown.edu/">SCML competition</a> provides the perfect opportunity to develop RL-based solutions to this problem. Other than the joy of solving a new problem, the competition provides monetary prizes, and even travel scholarships (for selected students) to attend <a href="https://www.aamas2024-conference.auckland.ac.nz/">AAMAS 2024</a>, one of the top AI conferences in the world.</li><li>Everything you need to start working on this problem is available at the <a href="https://scml.cs.brown.edu/">competition website</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=076a205cd1c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SCML Standard Challenge: Long Term Planning and Automated Negotiation]]></title>
            <link>https://yasserfarouk.medium.com/scml-standard-d1f882b4f08e?source=rss-511819d0237d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1f882b4f08e</guid>
            <category><![CDATA[ijcai]]></category>
            <category><![CDATA[competition]]></category>
            <category><![CDATA[automated-negotiations]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[supply-chain]]></category>
            <dc:creator><![CDATA[Yasser Mohammad]]></dc:creator>
            <pubDate>Mon, 17 May 2021 06:39:04 GMT</pubDate>
            <atom:updated>2021-05-17T11:16:56.638Z</atom:updated>
            <content:encoded><![CDATA[<p>In previous articles, I introduced the topics of <a href="https://yasserfarouk.medium.com/automated-netotiation-can-it-be-the-next-big-thing-b4840ea0f407">automated negotiations</a>, the International Automated Negotiating Agents Competition (<a href="http://web.tuat.ac.jp/~katfuji/ANAC2021/">ANAC</a>), and the <a href="https://yasserfarouk.medium.com/scml-negotiate-your-way-to-fornute-f99849408e66">Supply Chain Management League</a> which will run as part of the official competitions track of <a href="https://ijcai-21.org/competitions/">IJCAI 2021</a> — the top AI conference — this year. Winners will receive monetary prizes and will be announced during IJCAI.</p><p>The <a href="https://yasserfarouk.medium.com/scml-negotiate-your-way-to-fornute-f99849408e66">previous article</a> focused on the <a href="http://www.yasserm.com/scml/scml2021oneshot.pdf">SCML-OneShot</a> track. This article focuses on the <a href="http://www.yasserm.com/scml/scml2021.pdf">SCML-Standard</a> track, which aims to combine the challenge of concurrent negotiation with long-term planning.</p><p>The SCML standard track runs in the world depicted in this image:</p><figure><img alt="SCML World" src="https://cdn-images-1.medium.com/proxy/1*y97HkTAyfmvKcHfbd1-YSg.jpeg" /><figcaption>SCML World</figcaption></figure><p>Here we have a set of products and manufacturing processes. Each process receives one item of product and converts it into one item of product at some cost. The chain of products and processes shown in the image above is called the <em>production graph</em>.</p><p>In this world, we have a set of factories. Each factory can execute one and only one process at some cost <em>specific to this factory</em>, known to it and unknown to all others. Agents control factories and try to maximize their profit.</p><p>Demand in this market is driven by a set of <em>exogenous sell contracts</em> for the set of factories that can execute <em>the last process </em>and supply is provided by a set of <em>exogenous buy contracts</em> for the set of factories that can execute the first process.</p><h4>Main Differences between SCML-Standard and SCML-OneShot</h4><p>Up until now, the world of SCML-Standard and <a href="https://yasserfarouk.medium.com/scml-negotiate-your-way-to-fornute-f99849408e66">SCM-OneShot</a> seem almost the same with the single difference that the production graph can be deep (i.e. with more than 2 processes) in the standard game. This simple change implies that the agent must be developed to work anywhere in the chain. Notice that agents in the first and last level of the chain are facing a mirror image problem. They both have some exogenous contracts on one side and need to negotiate to <em>match</em> them on the other side while maximizing their profits. In a deeper chain (as in SCML-Standard), agents in the middle need to negotiate on both sides <em>simultaneously</em>. This gives us the first main difference between the two tracks:</p><blockquote>Agents need to work anywhere in the chain not only on the edges in SCML-Standard.</blockquote><p>While we are at it, let’s discuss a second difference. In SCML-OneShot, agents have exactly one exogenous contract at every step and <em>all exogenous contracts are binding</em>. Moreover, once two agents reach an agreement through negotiation, this agreement is <em>immediately binding</em>. In SCML-Standard, neither of these properties is true anymore. Here, all agreements (exogenous and negotiated) become binding <em>only</em> if both agents party to it confirm the agreement by <em>signing</em> it during a call to their sign_all_contracts() method. This is both a blessing and a curse.</p><p>Why? On one hand, it provides the agent with the opportunity to choose which contracts to sign to maximize their profit. For example, if on the last day, an agent (with a production cost of 1) has 5 agreements to buy its input product at 10 dollars each and 2 agreements to sell its output product at 12 dollars each, it can choose to sign only 2 of the buy contracts achieving a profit of 2 × (12−1−10) = 2 dollars. If agreements were immediately binding, the agent would have lost (3 × 10 − 2 × (12−1−10) = 28) dollars. Can you see why?</p><p>Ok, that is the blessing 😀, what is the curse 😢? The partner can also decide not to sign. This means that it is simply impossible to be sure about your profits. In the example above, the agent may decide to sign two buy contracts but both of these suppliers decide not to sign them leading to it receiving no inputs. On the other hand, the two consumers may decide to sign which means that there will be a shortfall and the agent will have to pay a penalty for not delivering!!!</p><blockquote>The fact that contracts are binding only after being signed not upon agreement is a mixed blessing. An unfavorable agreement is not a disaster but it is impossible to estimate profits exactly.</blockquote><p>Two other main differences between SCML-OneShot and SCML-Standard make<em> all the difference</em>: products do not perish, and delivery time is negotiable. The first means that agents can accumulate an inventory of either their input material or output product and the latter means that agents can negotiate about receiving/sending products in the future (not only today). This combination means that the agent cannot only think about today while negotiating but needs to consider the future. This is why <em>long-term planning</em> becomes a central issue in agent development in SCML-Standard.</p><blockquote>Long term planning is a central issue in SCML-Standard because products do not perish and delivery time is a negotiation issue.</blockquote><p>You can see a complete list of differences between the two tracks with the associated implications <a href="http://www.yasserm.com/scml/diffs2021.pdf">here</a>.</p><p>From now on, we will refer to SCML-Standard simply as SCML.</p><h4>Agent Anatomy</h4><p>For an agent to be successful in the SCML world, it needs to buy input materials through negotiation, manufacture them, then sell output products through negotiation. It is possible to structure our agent as a combination of three main strategies/components:</p><ol><li><strong>Trading Strategy</strong>: Deciding the quantity (and price) to buy and sell at every day. This component can employ two subcomponents:</li></ol><ul><li>a pre-negotiation component that decides the quantities/prices to negotiate about based on the prediction of future market behavior (<strong>trade prediction strategy</strong>) and partner behavior (<strong>partner behavior prediction strategy</strong>)</li><li>a post-negotiation component that decides what agreements to sign as contracts (<strong>signing strategy</strong>).</li></ul><ol><li><strong>Negotiation Control Strategy</strong>: This component is responsible for proactively request negotiations, responding to negotiation requests, and actually conducting concurrent negotiations. This component can further be divided into two subcomponents:</li></ol><ul><li>a pre-negotiation component that decides which negotiations to accept and which to engage in (<strong>negotiation manager</strong>)</li><li>The <strong>negotiation algorithm</strong> is used to carry out the negotiations decided by the negotiation manager.</li></ul><ol><li><strong>Production Strategy</strong>: Decides what to produce every day.</li></ol><p>This is a visual representation of these components. You can watch a detailed description of these components <a href="https://youtu.be/dzGWkTL46Jk">here</a>.</p><figure><img alt="Agent Anatomy in SCML-Standard" src="https://cdn-images-1.medium.com/proxy/1*IeDTgV3LmRUvRV_W_JoM0w.png" /></figure><p>.</p><p>You can develop your agent by just overriding the base SCML2020Agent class and overriding some or all of its callbacks. Consider, for example, the callback on_negotiation_success(). When you receive this callback from the simulator, you need to do whatever is necessary to update your trading, negotiation control, and production strategies. This is what we call a <strong>monolithic</strong> agent. Maybe you can just call three internal methods to do the necessary work:</p><ul><li>trading_on_negotiation_sucess</li><li>negotiation_on_negotiation_sucess</li><li>production_on_negotiation_sucess</li></ul><p>Other than the obvious explosion in method names and the number of methods, this approach makes it harder for <em>others</em> to reuse your code. What if someone likes your trading strategy but not your negotiation strategy and at the same time likes the negotiation strategy of a third participant but not their trading strategy? It will not be easy for them to reuse the parts they like. Moreover, if all of these methods are sharing state by changing the agent’s data attributes, it may not be even possible to do the combination.</p><p>You are free to organize your agent in this monolithic way but — for this post — we will assume that you would like to use a more modular approach. In the scml package, we use <strong>cooperative inheritance</strong> to achieve this goal.</p><p>The SCML platform provides several components that can be used to implement each of the three strategies and you can combine and match them as you like to generate new agents. The tutorials explain existing components, their relationships and how to use them. This post focuses on explaining the idea of <strong>cooperative inheritance</strong> used to organize these relationships.</p><h4>Cooperative Inheritance</h4><p>The idea of collaborative inheritance is to make it possible for users of a class to replace the behavior of any part of it simply by adding another base class. In Python, our main weapon to implement this pattern is super(). For a clear explanation of this use of super, refer to <a href="https://rhettinger.wordpress.com/2011/05/26/super-considered-super/">Hettinger’s post</a> and his <a href="https://youtu.be/EiOglTERPEo">excellent talk</a> on YouTube.</p><h4>Super and the MRO</h4><p>Let’s consider the following three classes:</p><pre>class BaseAgent:<br>  def init():<br>        print(&quot;Base initalized&quot;)<br><br>class Component1:<br>    def init():<br>        print(&quot;C1 initialized&quot;)<br><br>class Agent(Component1, BaseAgent):<br>    pass</pre><p>BaseClass, and Component1 have implementations of the same method (init). What really matters here is the method name (parameters need not match). The Agent class inherits from both of them. What happens if we call init:</p><pre>Agent().init()</pre><p>You will see that Component.init() is called because we can see the following printed:</p><pre>C1 initialized</pre><p>Nevertheless, BaseAgent.init() was not called. We can solve that easily by changing our code to the following:</p><pre>class Component1:<br>    def init():<br>        print(&quot;C1 initialized&quot;)<br>        super().init()<br><br>class Agent(Component1, BaseAgent):<br>    pass<br><br>Agent().init()</pre><p>Now we get:</p><pre>C1 initialized<br>Base initialized</pre><p>That makes sense. The call to super() was resolved to BaseAgent.init() which is now called after the print in Component1.init() as expected.</p><p>What will happen if we change the order during inheritance:</p><pre>class BadAgent(Agent, Component1):<br>    pass<br>BadAgent().init()</pre><p>Now we get:</p><pre>C1 initialized</pre><p>What just happened? The python interpreter was asked to find BadAgent.init(), it looked in the base classes <strong>in the order they were given when BadAgent was defined</strong>. It looked first for Agent.init() and found it so it did not even try to find Component1.init(). That is why we do not see Base intialized. Can you imagine what will happen if we add super().init() to BaseAgent itself?❓</p><p>In general, the python interpreter will use some predefined <strong>Message Resolution Order (MRO)</strong> to locate any attribute (method or otherwise) you try to access. The MRO is simply a list of class names that are checked to locate any attribute. You can find the MRO associated with any class by calling its mro() method. Here are the resulting MROs for Agent and BaseAgent.</p><ul><li>Agent → [Agent, <strong>Component1</strong>, BaseAgent, object]</li><li>BadAgent → [BadAgent, <strong>BaseAgent</strong>, Component1, object]</li></ul><p>The algorithm used to find this order is called <a href="https://en.wikipedia.org/wiki/C3_linearization"><strong>C3 linearization</strong></a> and it is guaranteed to generate a list from any inheritance graph (even with common base classes). Two very important features of the MRO that is guaranteed by C3 are:</p><ul><li>Classes that appear first in the inheritance list are checked first.</li><li>Children are checked before parents.</li></ul><p>There are several blogs and tutorials that describe how this whole thing hangs together. For our purposes, I will focus on one specific emergent behavior that we use in collaborative inheritance. Consider the following two classes:</p><pre>class A:<br>    def init(self):<br>        print(&quot;A&quot;, end=&quot;&quot;)<br><br>class B(A):<br>    def init(self):<br>        print(&quot;B&quot;, end=&quot;&quot;)<br>        super().init()</pre><p>It is easy to see that B().init() will print BA. But what if we want to change how init() behaves and force it to print BE <strong>without touching </strong><strong>B, or reimplementing </strong><strong>A.init()</strong>? Here is one way to do that based on our current understanding of super():</p><pre>class E(A):<br>    def init(self):<br>        print(&quot;E&quot;, end=&quot;&quot;)<br><br>class D(B, E):<br>    pass</pre><p>Let’s check the MRO. Because B appears before E in D’s inheritance list, it must appear before it in the MRO. Moreover, because E is a subclass of A, it must appear before it in the MRO. These constraints give us the following MRO: [<em>D</em>,<em>B</em>,<em>E</em>,<em>A</em>]. E comes before A which forces the call to super() in B to call E’s version instead of A’s version which is what we want.</p><p>You can check for yourself that if E does not inherit from A, we get [<em>D</em>,<em>B</em>,<em>A</em>,<em>E</em>] which will print BA (i.e. E and D are really useless in this case).</p><p>Now if we run D().init(), we get BE as we wanted. The recipe is simple: To replace some method m() that a class X inherits from some parent B (directly or indirectly):</p><ol><li>create a class E that inherits from B and implements m() <strong>without calling </strong><strong>super() in it</strong>.</li><li>create a class Y that inherits from X and E (in that order).</li></ol><p>Now Y behaves identically to X with the single exception that the method m() is now replaced with the new implementation in E.</p><p>To <strong>add behavior to</strong> some method m() that a class X inherits from some parent B (directly or indirectly):</p><ol><li>create a class E that inherits from B and implements m() <strong>and call </strong><strong>super() in it</strong>.</li><li>create a class Y that inherits from X and E (in that order).</li></ol><p>Now Y behaves identically to X with the single exception that the method B.m() is called either before or after E.m() depending on where did you call super() inside E.m(). As an example consider:</p><pre>class E1(A):<br>    def init(self):<br>        print(&quot;E&quot;, end=&quot;&quot;)<br>        super().init()<br><br>class D1(B, E):<br>    pass<br><br>class E2(A):<br>    def init(self):<br>        super().init()<br>        print(&quot;E&quot;, end=&quot;&quot;)<br><br>class D2(B, E):<br>    pass</pre><p>You can confirm that D1().init() prints BEA while D2().init() prints BAE.</p><blockquote>You can use inheritance to either <strong>add to</strong> or <strong>override</strong> the behavior of some method no matter how deep is it in the inheritance hierarchy using cooperative inheritance. We call <strong>adding</strong> behavior <strong>hooking into</strong> a method and <strong>overriding</strong> behavior (well) <strong>overriding</strong> a method.</blockquote><h4>Cooperative Inheritance in SCML</h4><figure><img alt="Cooperative Inheritance in SCML" src="https://cdn-images-1.medium.com/proxy/1*v1qqTpmOQzHrEtT0jTx4aQ.jpeg" /></figure><p>As I explained earlier, there are three main problems that the agent faces in SCML-Standard:</p><ul><li><strong>Trading Strategy</strong> Long-term planning.</li><li><strong>Negotiation Strategy</strong> concurrent negotiation to achieve the plan.</li><li><strong>Production Strategy</strong> controlling production in the factory.</li></ul><p>These are top-level strategies. To maximize reuse, we also provide lower-level strategies that are used by these top-level strategies. This makes it easier to choose and match. Some of the most important lower-level strategies are:</p><ul><li><strong>Signing Strategy</strong> Decides what to sign. Used usually by the trading strategy.</li><li><strong>Trade Prediction Strategy</strong> Predicts future trade. Used by the trading strategy.</li><li><strong>Execution Rate Prediction Strategy</strong> Predicts the probability that a given contract will actually be signed and executed. Used by both the negotiation manager and trading strategy.</li></ul><p>The scml package defines one class of strategies to match each of these top-level and lower-level strategies. Each of these is called a component and a component is defined by the following:</p><ul><li><strong>Hooks</strong> These are the callbacks that it hooks into. For example, the <strong>DemandDrivenProductionStrategy</strong> hooks into the callback <strong>on_contracts_finalized</strong> to determine what to produce based on knowledge of signed contracts. Because this is a hook, other components like say the trading strategy can hook into the same callback. This is another hook example from the same strategy:</li></ul><pre>class SupplyDrivenProductionStrategy(ProductionStrategy):<br>    def step(self): <br>        super().step() <br>        ...</pre><ul><li><strong>Overrides</strong> These are the callbacks that are overridden by the component. For example, some trading strategies override the callback sign_all_contracts to control what is being signed by the agent. This is how overriding is implemented for this case (notice that super() is not called):</li></ul><pre>class SignAll:<br>    def sign_all_contracts(self, contracts):<br>        return [self.id] * len(contracts)</pre><ul><li><strong>What it provides</strong> These are methods/attributes that the component controls and provides for other components (usually to read but not modify). For example, all trading strategies provide an attribute needed_inputs that defines the number of input product units needed per day. The negotiation manager may (and does) use this value to adjust its behavior during negotiation.</li><li><strong>What it requires</strong> These are methods/attributes that the component expects to exist in the agent. For example, the negotiation manager <strong>expects</strong> that the agent will have an attribute needed_inputs (see the previous point).</li><li><strong>Attributes</strong> These are named arguments passed to the constructor of the component and used to control its behavior. The constructor of the component should consume these parameters and pass the rest (through super()) to the constructors of other components. For example, the KeepOnlyGoodPrices signing strategy has two parameters buying_margin and selling_margin that define what a good price is. That is how its constructor is implemented:</li></ul><pre>def __init__(self, *args, buying_margin=0.5, selling_margin=0.5, **kwargs):<br>    super().__init__(*args, **kwargs) <br>    self._buying_factor, self._selling_factor = ( <br>        1.0 + buying_margin, 1.0 - selling_margin<br>    )</pre><p>As you can see, the constructor consumes these two arguments and passes the rest along to construct other components. This is how the system can construct all components without knowing a-priori the names/types of arguments needed to construct each component.</p><ul><li><strong>Abstract</strong> These are optional methods that all children must implement.</li></ul><p>Using these components, the built-in DecentralizingAgent is now extremely simple to write:</p><pre>class DecentralizingAgent(<br>    _NegotiationCallbacks,<br>    StepNegotiationManager,<br>    PredictionBasedTradingStrategy,<br>    SupplyDrivenProductionStrategy,<br>    SCML2020Agent,<br>):<br>    pass</pre><p>That is the <strong>complete code</strong>. It simply chooses which component to use for each of the three top-level strategies. The _NegotiationCallbacks implements the <em>abstract</em> methods needed by the specific negotiation manager used.</p><p>Here is another example:</p><pre>class IndDecentralizingAgent(<br>    _NegotiationCallbacks,<br>    IndependentNegotiationsManager,<br>    PredictionBasedTradingStrategy,<br>    SupplyDrivenProductionStrategy,<br>    SCML2020Agent,<br>):<br>    def create_ufun(self, is_seller: bool, issues=None, outcomes=None):<br>        if is_seller:<br>            return LinearUtilityFunction((1, 1, 10))<br>        return LinearUtilityFunction((1, -1, -10))</pre><p>That is the complete code of another agent. You can immediately see that this agent shares the trading and production strategies with the previous agent. It only differs in the negotiation manager it uses which has one extra <em>abstract</em> method (create_ufun()) that is implemented by this agent.</p><p>In 2021, the game introduced public information about the market that was not available to the agent in 2020. To build a similar agent that utilizes this information, we only had to add:</p><pre>class MarketAwareIndDecentralizingAgent(<br>    KeepOnlyGoodPrices,<br>    MarketAwareTradePredictionStrategy,<br>    IndDecentralizingAgent,<br>):<br>    pass</pre><p>Here you can see that two components are being replaced. Both components are lower-level components (the signing and trade prediction strategies). Now this agent can utilize the extra information without us touching the code of the trading strategy itself.</p><h4>Summary</h4><p>The main goal of this post was to clarify the main differences between the SCML-Standard and SCML-OneShot games and provide the justification and some overview of cooperative inheritance as used in SCML-Standard. The next step is to check the <a href="http://www.yasserm.com/scml/scml2020docs/tutorials.html">tutorials</a> for detailed explanations of each component and examples of creating new agents based on them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1f882b4f08e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SCML: Negotiate Your Way to Fornute]]></title>
            <link>https://yasserfarouk.medium.com/scml-negotiate-your-way-to-fornute-f99849408e66?source=rss-511819d0237d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f99849408e66</guid>
            <category><![CDATA[game-theory]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[competitions-online]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Yasser Mohammad]]></dc:creator>
            <pubDate>Sat, 01 May 2021 08:19:53 GMT</pubDate>
            <atom:updated>2021-05-01T11:12:12.220Z</atom:updated>
            <content:encoded><![CDATA[<p>In a <a href="https://yasserfarouk.medium.com/automated-netotiation-can-it-be-the-next-big-thing-b4840ea0f407">previous article</a>, I introduced the topic of automated negotiations and the International Automated Negotiating Agents Competition (<a href="http://web.tuat.ac.jp/~katfuji/ANAC2021/">ANAC</a>) which will run as part of the official competitions track of <a href="https://ijcai-21.org/competitions/">IJCAI 2021</a> — one of the top AI conferences — this year. Winners will receive monetary prizes and will be announced during IJCAI.</p><p>This article focuses on the Supply Chain Management League (<a href="https://scml.cs.brown.edu/">SCML</a>) which runs as part of this competition.</p><p>SCML has three tracks this year:</p><ul><li><strong>OneShot</strong> A track focused on the problem of repeated concurrent negotiation.</li><li><strong>Standard</strong> A track that combines the concurrent negotiation problem with effective long-term planning.</li><li><strong>Collusion</strong> A track that focuses on methods that businesses can employ to <em>corner</em> the market.</li></ul><p><em>OneShot</em> is the simplest of these three tracks to explain. This article will focus exclusively on the SCML-OneShot game. In future articles, I will explain the main differences between the three tracks.</p><p>An overview of the SCML-OneShot game is available <a href="http://www.yasserm.com/scml/overview_oneshot.pdf">here</a> and a full description for the details-savvy person is available <a href="http://www.yasserm.com/scml/scml2021oneshot.pdf">here</a>.</p><blockquote>SCML has an online competition which allows you to compete against other participants <em>before</em> the official competition. Last year, this proved invaluable for improving agents and removing the worst of their bugs and unexpected behaviors by the time of the official competition.</blockquote><h4>A Simple-Enough Business-Like Game</h4><blockquote>Everything should be made as simple as possible, but no simpler, <em>Albert Einstein</em>.</blockquote><p>The game simulates a common business environment in which factories (capable of running predefined manufacturing processes) try to maximize their profits through trade. There are exactly 3 products and 2 processes as shown in the figure. Each agent controls exactly one factory. <em>All trade within the system is done through closed bilateral negotiations between agents and their suppliers</em>. The market is driven by a set of <em>exogenous</em> contracts (non-negotiable) generated by the system every day (More on that later).</p><p><strong>Game Participants develop these agents intending to maximize their profits in this competitive market.</strong></p><figure><img alt="The SCM world: See competition website" src="https://cdn-images-1.medium.com/proxy/1*wNvAFSbNOx_bU_dh6Fk5SA.jpeg" /><figcaption>The SCM world: See <a href="https://scml.cs.brown.edu/">competition website</a></figcaption></figure><p>Agents that receive the raw material and generate the intermediate product are called <em>L</em>0 agents. Agents that receive the intermediate product and generate the final product are called <em>L</em>1 agents. The product type consumed by the factory controlled by an agent is called its <em>input product</em> and the product it produces is called its <em>output product</em>.</p><h4>The simulation and utility function</h4><p>Every day (simulation step), each <em>L</em>0 factory receives one exogenous contract specifying a quantity and a unit price (supplies of the raw material) for this day and each <em>L</em>1 factory receives one exogenous contract specifying a quantity and a unit price of the final product (sales) for the same day. Production and transportation are assumed to take no time.</p><p><em>L</em>0 and <em>L</em>1 agents need to negotiate together to secure <em>intermediate product</em> contracts to use their supplies (for <em>L</em>0 agents) or satisfy their sale obligations (for <em>L</em>1 agents). All products that remain unsold at the end of the day perish (i.e. has no value). Moreover, agents pay a disposal cost for any input products they buy and never sell and pay a shortfall penalty for any sales they cannot satisfy.</p><p>Exogenous contracts drive the market. The profit of an agent can be calculated using a simple formula:</p><pre>profit = revenue - supply cost - production cost - shortfall penalty - disposal<br>cost</pre><p>Revenue comes from sales and is simply the total price of sell contracts executed by the agent (<em>qout</em> × <em>pout</em>). There are four sources of expenditure that <em>should</em> be minimized:</p><ol><li><strong>Supply Cost</strong> The total price of all buy contracts (<em>qint</em> × <em>qin</em>).</li><li><strong>Production Cost</strong> The production cost per unit multiplied by the input quantity.</li><li><strong>Shortfall Penalty</strong> A penalty paid for each item in the sell contracts not honored by the agent. For example, if the agent has an output contract with 5 but could only produce 3 items, it will pay this penalty for the remaining two (besides not receiving their price obviously).</li><li><strong>disposal penalty</strong> A penalty paid for each item bought but not manufactured by the agent.</li></ol><p>Considering this profit calculation for a moment, it is clear that the core problem facing the agent is to <em>match its input and output quantities while maximizing the difference in their unit prices (for its advantage)</em>. Your goal as an agent designer is to maximize your overall profit over the simulation time.</p><p><strong>The core problem targeted by SCML-OneShot is</strong> <em>repeated concurrent negotiation</em> <strong>with a known — yet variable — utility function.</strong></p><h4>Why is SCML-OneShot interesting?</h4><p>SCML-OneShot is an <a href="https://en.wikipedia.org/wiki/Complete_information#Complete_versus_perfect_information"><strong>incomplete information game</strong></a> because the payoffs (utility functions) are not common knowledge.</p><p>Moreover, automated negotiation in general is <em>not a zero-sum game</em>. This combination of a <strong>nonzero-sum incomplete information</strong> game is what makes SCML-OneShot interesting from the point of view of game theory.</p><p>Machine learning can be used in many ways in this game:</p><ul><li>Learning about your partners. For example, try to estimate the utility function of your partner based on market conditions, past interactions with it, and its financial status.</li><li>Learning how to negotiate. An obvious method here is to employ RL with either self-play or playing against built-in agents provided by the organizers. You can then either adapt your learned policy in real-time or just use it as it is.</li><li>Learning to predict market conditions. For example, the changes in trading prices of the intermediate product can be important in devising an effective negotiation strategy.</li></ul><p>We tried to set things up to make applying ML both effective and challenging:</p><ul><li>The distribution of exogenous contracts between agents is not completely random, leading to similar utility functions for each agent on all days. This facilitates learning about partners.</li><li>The total demand in the market (total exogenous contract quantities and average prices) are common knowledge. This simplifies reasoning about the game and provides context information for ML methods.</li><li>The probability distributions from which game parameters are sampled are all described in the <a href="http://www.yasserm.com/scml/scml2021oneshot.pdf">full game description</a>.</li></ul><h4>How can I get started with SCML?</h4><p>To get started, you need to install the <a href="https://www.github.com/yasserfarouk/scml">scml</a> package from PyPI:</p><pre>pip install scml</pre><p>You can then download the <a href="http://www.yasserm.com/scml/oneshot.zip">skeleton</a> from the <a href="https://scml.cs.brown.edu/">competition website</a>.</p><p>Moreover, there is an optional visualization package scml-vis that helps you <strong>see</strong> what happens in simulations and tournaments. It is also pip installable:</p><pre>pip install scml-vis</pre><p>And you can use it to explore logs from game simulations:</p><figure><img alt="SCML-OneShot example visualization charts using scml-vis" src="https://cdn-images-1.medium.com/proxy/1*0MA0q1dOBU5IEZUjlZhSnQ.png" /><figcaption>SCML-OneShot example visualization charts using scml-vis</figcaption></figure><h4>How difficult is it to develop an agent?</h4><p>To develop an agent, you need to override two methods propose and respond for proposing and responding to offers from other agents. That is all folks 😎</p><p>Here is a complete agent. It is 34 lines of code.</p><pre>class GreedyOneShotAgent(OneShotAgent):<br>    &quot;&quot;&quot;A greedy agent based on OneShotAgent&quot;&quot;&quot;<br><br>    def init(self):<br>        self.secured = 0<br><br>    def step(self):<br>        self.secured = 0<br><br>    def on_negotiation_success(self, contract, mechanism):<br>        self.secured += contract.agreement[&quot;quantity&quot;]<br><br>    def propose(self, negotiator_id: str, state) -&gt; &quot;Outcome&quot;:<br>        return self.best_offer(negotiator_id)<br><br>    def respond(self, negotiator_id, state, offer):<br>        my_needs = self._needed(negotiator_id)<br>        if my_needs &lt;= 0:<br>            return ResponseType.END_NEGOTIATION<br>        return (<br>            ResponseType.ACCEPT_OFFER<br>            if offer[QUANTITY] &lt;= my_needs<br>            else ResponseType.REJECT_OFFER<br>        )<br><br>    def best_offer(self, negotiator_id):<br>        my_needs = self._needed(negotiator_id)<br>        if my_needs &lt;= 0:<br>            return None<br>        ami = self.get_ami(negotiator_id)<br>        if not ami:<br>            return None<br>        quantity_issue = ami.issues[QUANTITY]<br>        unit_price_issue = ami.issues[UNIT_PRICE]<br>        offer = [-1] * 3<br>        offer[QUANTITY] = max(<br>            min(my_needs, quantity_issue.max_value),<br>            quantity_issue.min_value<br>        )<br>        offer[TIME] = self.awi.current_step<br>        if self._is_selling(ami):<br>            offer[UNIT_PRICE] = unit_price_issue.max_value<br>        else:<br>            offer[UNIT_PRICE] = unit_price_issue.min_value<br>        return tuple(offer)<br><br>    def _needed(self, negotiator_id=None):<br>        return self.awi.current_exogenous_input_quantity + \<br>               self.awi.current_exogenous_output_quantity - \<br>               self.secured<br><br>    def _is_selling(self, ami):<br>        return ami.annotation[&quot;product&quot;] == self.awi.my_output_product<br><br>world, ascores, tscores = try_agent(GreedyOneShotAgent)</pre><p>The main idea of this agent is pretty simple. It tries to <em>secure</em> as much of its needs (sales/supplies) as possible in every negotiation at the best possible price for itself.</p><p>To achieve this goal, the agent keeps track of the quantity it secured in its init, step and on_negotiation_success callbacks.</p><pre>def init(self):<br>    self.secured = 0<br><br>def step(self):<br>    self.secured = 0<br><br>def on_negotiation_success(self, contract, mechanism):<br>    self.secured += contract.agreement[&quot;quantity&quot;]</pre><p>Moreover, it defines a helper that calculates the amount it needs by subtracting the exogenous quantity it has from the amount it secured:</p><pre>def _needed(self):<br>    return self.awi.current_exogenous_input_quantity + \<br>           self.awi.current_exogenous_output_quantity - \<br>           self.secured</pre><p>Notice that either the exogenous input quantity or the exogenous output quantity (or both) will always be zero. Now that the agent can calculate how much it needs to buy/sell, it implements the negotiation related call-backs (propose and respond).</p><p>Here is the full implementation of propose:</p><pre>def propose(self, negotiator_id: str, state) -&gt; &quot;Outcome&quot;:<br>        return self.best_offer(negotiator_id)</pre><p>The agent is always offering its best offer which is calculated in the best_offer method to be discussed later. It does not concede at all.</p><p>Responding to opponent offers is also simple:</p><ul><li>it starts by calculating its needs using the helper neededand ends the negotiation if it needs no more sales/supplies</li></ul><pre>my_needs = self._needed()<br>    if my_needs &lt;= 0:<br>        return ResponseType.END_NEGOTIATION</pre><ul><li>If the offered quantity is less than its needs, accept the offer. Otherwise reject the offer.</li></ul><pre>return ResponseType.ACCEPT_OFFER if offer[QUANTITY] &lt;= my_needs else ResponseType.REJECT_OFFER</pre><p>Most of the code is in the best_offer method which calculates the best offer for a negotiation <em>given the agreements reached so far</em>. Let’s check it line by line:</p><ul><li>The agent checks its needs and returns None ending the negotiation if it needs no more sales/supplies. We also get access to the AMI.</li></ul><pre>my_needs = self._needed()<br>    if my_needs &lt;= 0:<br>        return None<br>    ami = self.get_ami(negotiator_id)<br>    if not ami:<br>        return None</pre><ul><li>It then finds out the Issue objects corresponding to the quantity and unit-price for this negotiation and initializes an offer (we have 3 issues)</li></ul><pre>quantity_issue = ami.issues[QUANTITY]<br>    unit_price_issue = ami.issues[UNIT_PRICE]<br>    offer = [-1] * 3</pre><ul><li>The time is always the current step.</li></ul><pre>offer[TIME] = self.awi.current_step</pre><ul><li>The quantity to offer is simply the needs of the agent without mapped within the range of the quantities in the negotiation agenda (note that this may lead the agent to buy more than its needs).</li></ul><pre>offer[QUANTITY] = max(<br>        min(my_needs, quantity_issue.max_value), quantity_issue.min_value<br>    )</pre><ul><li>Finally, the unit price is the maximum possible unit price if the agent is selling otherwise it is the minimum possible price. Note that is_selling() assumes that the agent will never find itself in a middle layer in a deep negotiation. We will alleviate this issue later.</li></ul><pre>if self._is_selling(ami):<br>        offer[UNIT_PRICE] = unit_price_issue.max_value<br>    else:<br>        offer[UNIT_PRICE] = unit_price_issue.min_value<br>    return tuple(offer)</pre><p>This agent is extremely simplistic in several ways:</p><ol><li>It insists on its price all the time making it too inflexible.</li><li>It accepts offers <em>without even considering the price</em> which makes it too gullible.</li><li>It does not even try to learn about its partners’ behavior during negotiation or over consecutive days.</li></ol><p>To see examples of better agents, check the <a href="http://www.yasserm.com/scml/scml2020docs/tutorials/02.develop_agent_scml2020_oneshot.html">official tutorials</a></p><h4>Online Competition and Available Resources</h4><p>The online SCML competition is already open and will run until the beginning of July 2021. You can find all the details about the game, how to participate, and a wealth of tutorials and other resources on the <a href="https://scml.cs.brown.edu/">competition website hosted by Brown University</a>.</p><h4>Summary</h4><p>This article introduced the SCML-OneShot game which provides an interesting and novel challenge for the AI community. The competition (which runs as part of ANAC @ IJCAI 2021) is open to all.</p><p>Consider submitting to SCML and negotiate your way to fortune. Sorry!! Make your agent do that for you …</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f99849408e66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Negotiation: Can it be the next big thing?]]></title>
            <link>https://yasserfarouk.medium.com/automated-netotiation-can-it-be-the-next-big-thing-b4840ea0f407?source=rss-511819d0237d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4840ea0f407</guid>
            <category><![CDATA[competitions-online]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[negotiation]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Yasser Mohammad]]></dc:creator>
            <pubDate>Sat, 17 Apr 2021 03:29:32 GMT</pubDate>
            <atom:updated>2021-04-17T03:31:25.543Z</atom:updated>
            <content:encoded><![CDATA[<p>Negotiation shares one feature with Christmas and love, it is everywhere. We <em>negotiate</em> our way when passing the street, businesses negotiate constantly to maximize their profits and in search for win-win agreements, and haggling over prices is a constant annoyance with which some of us have a love-hate relationship.</p><blockquote>In this article, I will try to convince you that automated negotiation can be the next big thing in AI and invite you to participate in the leading automated negotiation competition in the world (ANAC).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*sIzLWHGJzkOH5u7ar_cNGw.jpeg" /><figcaption>Automated Negotiation</figcaption></figure><p>With the increased adoption of AI technologies in industrial and business applications, comes renewed interest in automated negotiation as a possible way to coordinate the behavior of a plethora of automated agents representing different players in markets and industrial applications. This can clearly be sen in the rise of startups like <a href="https://pactum.com/">Pactum</a> dedicated to AI-based automated negotiation, interest from IT giants like <a href="https://www.nec.com/en/press/201908/global_20190821_02.html">NEC</a> as well as industrial organizations like the <a href="https://www.iiconsortium.org/">IIC</a> industrial consortium, let alone international organizations like <a href="https://uncefact.unece.org/display/uncefactpublic/eNegotiation">UN/CEFACT</a>.</p><p>The earliest concrete result in automated negotiation I could find is the <a href="https://www.jstor.org/stable/pdf/1907266.pdf?casa_token=OKboGSZB6HEAAAAA:9RuBO0wqIUJa2HxmW93yEvJ-6IGRM_oeW0yd3pJZpN-AXPgwEhL8O2Mwj2V4aoq1XW5iAM1V8rZBlnYoNr3kcPc05Dl9SPG5ZUX5Hilz1Dhqv8CGHkZP">Nash solution for the bargaining problem</a> introduced by Nash (of beautiful minds fame) in 1950. Economists, game theorists, cognitive scientists and AI researchers, all contributed to our current understanding of automated negotiation.</p><h4>What is automated negotiation?</h4><p>But, what is automated negotiation? Informally, it is a process by which self-interested agents try to achieve agreements that are beneficial for both of them through a process (called a negotiation protocol or mechanism) with fully defined rules. When one or more of the negotiators is a computer program (agent), the process is called automated negotiations. When one of the negotiation partners is human, psychological factors like priming effects, the availability heuristic and focus points become the bread and butter of negotiators.</p><blockquote>We focus here in the fully automated case in which all partners are AI agents.</blockquote><p>To make our discussion more tangible, we will use an opens-source negotiation library called <a href="http://www.yasserm.com/negmas">NegMAS</a> (I am a maintainer of this library so this is a plug). You can install negmas using:</p><pre>pip install negmas</pre><p>If you prefer to live in Java-Land, you can use the excellent <a href="http://ii.tudelft.nl/genius/">Genius</a> or <a href="https://tracinsy.ewi.tudelft.nl/pubtrac/GeniusWeb">GeniusWeb</a> libraries.</p><p>Negotiation happens between self-interested agents about some pre-defined <strong>negotiation issues</strong>. A common example is a negotiation between a supplier and buyer about a contract. Negotiation issues in this case may include <em>price</em>, <em>quantity</em>, <em>delivery schedule</em>, <em>penalties</em>, <em>quality clauses</em>, etc. Each possible assignment of a value to each negotiation issue (from its domain) is called an <strong>outcome</strong>. The set of all possible outcomes is called the <strong>outcome-space</strong>.</p><p>In NegMAS, you can define an outcome-space by simply specifying the issues:</p><pre>issues = [Issue(10, &quot;quantity&quot;), Issue((0.0, 1.0), name=&quot;price&quot;),<br>Issue([&quot;tomorrow&quot;, &quot;dayafter&quot;], &quot;delivery&quot;)]</pre><p>Outcomes are represented as either lists or dicts:</p><pre>o1 = (5, 0.5, &quot;tomorrow&quot;)<br>o2 = dict(quantity=5, price=0.5, delivery=&quot;tomorrow&quot;)</pre><p>NegMAS provides several methods for manipulating and using outcome-spaces (e.g. sampling, enumeration, validity checks, etc).</p><p>The preferences of an agent are usually represented by a <strong>utility function</strong> which maps outcomes to real values representing <strong>how good</strong> is an outcome for the agent. Common types of utility functions are linear aggregation utility functions, hyper-rectangle nonlinear utility functions, among many others.</p><p>In NegMAS, a utility function is represented by a UtilityFunction object. It is also possible to represent other types of preferences (e.g. partial orderings over outcomes). In our running example, we can define the utility function of the seller as:</p><pre>seller_util = LinearUtilityAggregationFunction(<br>    dict(<br>        price=lambda x: x, <br>        quantity= lambda x: 0.5 * x, <br>        delivery= dict(tomorrow= 1.0, dayafter= 0.0)<br>    ),<br>    weights = [1.0, 2.0, 0.5]<br>)</pre><p>Agents negotiate together by following a <strong>negotiation protocol</strong> (sometimes it is called <strong>negotiation mechanism</strong>) which specifies what each agent can and cannot do at every step of the negotiation.</p><p>The simplest negotiation mechanism is the alternating offers protocol in which agents take turns offering outcomes until one gets accepted, an agent leaves the negotiation or a timeout condition is met.</p><p>For more details on how to actually implement negotiators, mechanisms and conduct negotiation sessions, check <a href="http://www.yasserm.com/negmas">NegMAS Documentation</a></p><h4>Why is automated negotiation interesting for AI?</h4><p>Now that we have some idea about what negotiation is and how to run automated negotiations, why should anyone care?</p><p>Automated Negotiation is a <strong>game</strong> <em>in the academic game-theoretic sense</em>. It can be compared with Chess, Poker, or Go but it provides a somewhat unique challenge.</p><p>Most of these classic games were zero-sum games in the sense that one player’s gains are the other player’s losses. Some other games considered by game-theory experts are <em>collaborative games</em> that can lead to win-win situations.</p><p><strong>Negotiation on the other hand is a game with many faces</strong>. Depending on the relationship between the utility functions of the players (partners), it can be a zero-sum game, a collaborative game or anything else. <strong>The most important point here is that the agent does not know what type of game is it in</strong>.</p><p>This means that a general automated negotiation agent should be able to maximize its expected utility (or minimize its regret) in a wide spectrum of negotiation scenarios corresponding to a wide variety of games with different competitiveness.</p><p>This challenge of playing a game with incomplete information is not unique to automated negotiation. Nevertheless, I argue that automated negotiation provides a concrete case that can be adjusted in complexity by simple modifications to the scenario constraints leading to a common workbench for developing <strong>general AI</strong> approaches.</p><p>For example, we can consider negotiations with uncertainty on the agent’s own utility function, negotiations in which the agent can reduce this uncertainty at a cost, concurrent inter-related negotiations, negotiations with non-stationary utility functions, repeated negotiations, …...</p><p>Automated negotiations provides the unique combination of a practically applicable problem that drips with opportunities for new algorithmic innovations.</p><h4>The ANAC Competition</h4><p>AI likes competitions. Many of the most important advances in AI came as a response of a well designed game. In 2012, deep-learning registered in everyone’s radar after it achieved super human performance in a pattern recognition <strong>competition</strong>, and won the ICDAR Chinese handwriting <strong>competition</strong>.</p><p>Some researchers have seen the potential of automated negotiations for a long time now. A small research community is thriving around the International Automated Negotiating Agents Competition (<a href="http://web.tuat.ac.jp/~katfuji/ANAC2021/">ANAC</a>) since 2010.</p><p>The competition runs <em>every year</em> (for 12 incarnations now) with one of the top AI conferences in the world. It started in conjunction with <a href="https://aamas2021.soton.ac.uk/">AAMAS</a> but since 2017 is running on the official competition track of <a href="https://ijcai-21.org/competitions/">IJCAI</a>.</p><p>This year, ANAC has five different challenges:</p><ul><li>Automated Negotiation League, exploring strategies for building effective general purpose negotiation agents.</li><li>Human-Agent League, exploring strategies for negotiating with people using natural interaction modalities and capable of displaying and manipulating emotions.</li><li>Supply Chain Management League, exploring strategies for negotiation in a business-like simulation environment focusing on concurrent negotiation, long term trading and learning from repeated encounters.</li><li>Werewolf Game League, focusing on the ability to build consensus, identify agents of opposing teams and coordinating voting them out of the game.</li><li>HUMAINE League, exploring competitive negotiation in an immersive environment.</li></ul><p>Consider automated negotiation for your next project. It may be where we get the next big thing in AI, business or both.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4840ea0f407" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SCML: A new AI competition]]></title>
            <link>https://yasserfarouk.medium.com/scml-a-new-ai-competition-64a80801fa12?source=rss-511819d0237d------2</link>
            <guid isPermaLink="false">https://medium.com/p/64a80801fa12</guid>
            <category><![CDATA[negotiation]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Yasser Mohammad]]></dc:creator>
            <pubDate>Mon, 09 Mar 2020 12:27:48 GMT</pubDate>
            <atom:updated>2020-03-09T12:27:48.454Z</atom:updated>
            <content:encoded><![CDATA[<p>Negotiation is ubiquitous in our societies. With the wide-spread adoption of AI by businesses, internal operations of business units are increasingly being automated and automated negotiation between agents representing these business units is a promising direction that is starting to gain the interest of academics and industrialists. Automated negotiations promise faster more efficient agreements and more win-win deals to everyone.</p><p>The Automated Negotiation Agents Competition (ANAC) started in 2010 in conjunction with the eights International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2010) with the aim of advancing the state of the art in automated negotiation research. The competition succeeded in generating novel research in agent design and expanded to encompass several leagues focusing on different subproblems of automated negotiation. Since 2019, ANAC is run in conjunction with the International Joint Conference on Artificial Intelligence one of the largest AI conferences on the planet.</p><p>This article introduces the Supply Chain Management League (SCML), one of the five official leagues of ANAC 2020 and one of the newest additions to the competition.</p><h4>The SCM World</h4><p>Competitors in the SCM league are required to build an agent that controls a factory embedded in a simulated market. Besides controlling production, the agent should negotiate with other agents representing suppliers and consumers to secure supplies and sales. The score of a competitor is the median profit accumulated by all instantiations of her agent in thousands of simulations with varying configurations.</p><p>Let’s unwrap this description a bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*B_rHizKFJMuFooqh" /><figcaption>Overview</figcaption></figure><p>The figure above shows a typical SCM world with four products organized in a chain. Each factory is capable of running one and only one of the three manufacturing processes that exist in this world (shown with the same color as the process). Each agent in the system controls one of these factories and is responsible for buying the input material, running production to generate the output product and selling output materials. All buying and selling is conducted through negotiation with two exceptions:</p><ul><li>The factories at the first level (green factories) have predefined contracts for <em>buying</em> the raw material (silicon in this case). These contracts are revealed over the simulation time and provide the <em>supply</em>.</li><li>The factories at the last level (black factories) have predefined contracts for <em>selling</em> the final product (printed boards). These contracts are, again, revealed over the simulation time and provide the <em>demand</em>.</li></ul><p>The system is <em>event-driven</em> where the agent is implemented as responses to events (e.g. a negotiation request is received, a contract is concluded, etc).</p><h4>Why should you care?</h4><p>The SCML provides a novel challenge for those interested in AI. It is not just another supervised learning task that just needs more computation power thrown at it (even though it can have that). There are several points at which you can make a contribution that results in winning:</p><ul><li>The overall strategy of the agent: what should the agent trade and when? What should be its <em>goals</em> while negotiating? Under which circumstances should it refrain from trading?</li><li>The negotiation strategy: with whom should the agent negotiate and how?</li><li>The production strategy: How can the agent control production so it does not over-produce while keeping all its obligations?</li></ul><p>Given that this is the second year of the competition, there are still several low-hanging fruits. If you are building a portfolio of AI projects, this competition provides a unique challenge. If you are looking for a research topic in multi-agent systems, this competition provides several interesting problems that are of value to real businesses in the real-world.</p><h4>Agent Anatomy</h4><p>The organizers of SCML provide several built-in agents as examples of full implementations with varying levels of success. A major decision in designing an agent for SCML is to divide up the work that needs to be done into reusable chunks that can be reasoned about as independently as possible. The decomposition used by all built-in strategies in SCML (which you reuse) is a three ways decomposition into the following components:</p><ol><li><strong>Trading Strategy</strong> This is the central planner/CEO of the system. It decides what needs to be bought and sold and keeps track of the fraction of this that is already secured. It can use a <strong>Prediction Strategy</strong> for predicting supply and demand. Given that the built-in strategies are simplistic, this is one obvious target of development.</li><li><strong>Negotiation Manager</strong> This is the sales and supplies acquisition component. It is responsible of meeting the schedules set by the <em>trading strategy</em> through negotiation. Internally, it uses a <strong>Negotiation Strategy</strong> to decide how to negotiate, and a <strong>Signing Strategy</strong> to decide which of the agreements reached by its subordinate negotiators should be signed into binding contracts. Again, existing signing and negotiation strategies are simple and provide another obvious target of development.</li><li><strong>Production Strategy</strong> This is the manufacturing controller of the agent. It decides what should be produced when in light of the <em>trading strategy</em>.</li></ol><h4>Developing Your Agent</h4><p>To start developing your agent, you need to install the <a href="https://github.com/yasserfarouk/scml">SCML package</a> using pip</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/affc70846cd83f6037ce304c92656de1/href">https://medium.com/media/affc70846cd83f6037ce304c92656de1/href</a></iframe><p>There are step by step <a href="http://www.yasserm.com/scml/scml2020docs/tutorials.html">tutorials</a> that cover all stages of development as well as <a href="https://www.youtube.com/playlist?list=PLqvs51K2Mb8IJe5Yz5jmYrRAwvIpGU2nF">video tutorials</a> if you prefer that.</p><p>There is <a href="http://www.yasserm.com/scml/scml2020docs/tutorials/02.develop_agent_scml2020.html">a step-by-step tutorial</a> describing how to develop an agent for SCML.</p><p>I am not going to repeat those here.</p><p>Agents can be developed using <em>collaborative inheritance</em> which relies on the specific way super() works in python. Different components of the agent can <em>hook into</em> any of the callbacks and decide whether to allow other components to use the same callback (by calling super()) or to prevent them from doing so. There is an excellent <a href="https://rhettinger.wordpress.com/2011/05/26/super-considered-super/">blog</a> and <a href="https://www.youtube.com/watch?v=EiOglTERPEo">youtube video</a> by Raymond Hettinger about this way of code reuse.</p><p>Let’s see how can we develop an agent that uses almost all of the components of the DecentralizingAgent built-in agent and only replaces its simplistic <em>prediction strategy</em> with another one (that is not much better :-) ):</p><p>We start by importing what we need.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/644bce2977f3ec38276562edfcfe6259/href">https://medium.com/media/644bce2977f3ec38276562edfcfe6259/href</a></iframe><p>This is the full code of our agent:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b57a692cd645ac7e8cccd2a64dff949/href">https://medium.com/media/4b57a692cd645ac7e8cccd2a64dff949/href</a></iframe><p>For this agent, we will only override the trade prediction strategy used by the default trading policy of the DecentralizingAgent. We do that by overriding the only abstract method of the TradePredictionStrategy.</p><p>We can check how our strategy is doing compared with the built-in DecentralizingAgent by running a tournament between them. This is as easy as running the following two lines of code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/217f3545703ad19f11b0ceb91a6d6050/href">https://medium.com/media/217f3545703ad19f11b0ceb91a6d6050/href</a></iframe><p>Note that if you try this it may take several minutes (or hours) because it runs thousands of simulations to ensure the statistical significance of the results reported.</p><p>In my case, the winner was MyAgent achieving a median score of $0.34$ compared with $0.26$ for the built-in DecentralizingAgent. The difference was not statistically significant according to the automatically run ks-test ($p=0.3$). You can just replace this simple prediction model with a more accurate one (may be using supervised learning from a corpus of simulations you run offline) and it should improve the score of your agent.</p><h3>The Live Competition</h3><p>One unique feature of SCML is its <a href="https://scml.cs.brown.edu/">live competition</a> which allows competitors to submit their agent <em>at any time</em> and runs tournaments between them to give each competitor some feedback about her performance against the same opponents she will face in the official competition. The scores in the live competition will not affect the final scores of the official competition so it does not hurt to submit your agent even early during its development. Submit frequently to get more timely feedback about your ideas.</p><p>Not only does this help you as a developer decide what is working and what is not working for you but it is also much more fun compared with just submitting your agent and waiting for some email a month or so later informing you of your score.</p><p>The <a href="https://scml.cs.brown.edu/">live competition</a> provides everything you need to build your agent with pointers to documentation, tutorials, FAQs and more.</p><p><a href="https://scml.cs.brown.edu/register">Register</a> today and have your agent negotiate its way to the top.</p><p>In a future post, I will go deeper into different components of the agent, their interaction and the most obvious targets for improvement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64a80801fa12" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>