<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Will Pearson on Medium]]></title>
        <description><![CDATA[Stories by Will Pearson on Medium]]></description>
        <link>https://medium.com/@eb4890?source=rss-14885c3a9830------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Will Pearson on Medium</title>
            <link>https://medium.com/@eb4890?source=rss-14885c3a9830------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:41:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eb4890/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Translucency in AGI development]]></title>
            <link>https://medium.com/@eb4890/translucency-in-agi-development-206db4627bcd?source=rss-14885c3a9830------2</link>
            <guid isPermaLink="false">https://medium.com/p/206db4627bcd</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Will Pearson]]></dc:creator>
            <pubDate>Thu, 19 Apr 2018 21:41:25 GMT</pubDate>
            <atom:updated>2018-04-19T22:41:15.287Z</atom:updated>
            <content:encoded><![CDATA[<p>There are many risks around AGI development. From bad actors getting a hold of AGI and causing catastrophic disasters, to arms races between competing powers. A fair number of them are detailed in the <a href="https://maliciousaireport.com/">malicious ai report</a>. This means you might want to not release information about your systems.</p><p>However the AGI community needs to be able to share progress with different people for the following reasons.</p><ul><li>Responsible disclosure: Telling people that your AGI system might make their defences obsolete or cause large scale economic troubles and giving them some time and help to prevent problems before any release.</li><li>Helping other teams: There will be teams that share similar ideals of avoiding conflict and benefiting humanity etc, you want to preferentially help those teams.</li><li>Regulators: Regulators need to be able to see what the team is doing. However this is a position of trust. Bad actors might try to get into a regulatory position and get access to all the code of all the teams.</li><li>Informing the public: There is rightly lots of fear around AGI development. Secretive development will likely fuel fear and misunderstanding. Some amount of transparency will enable people to learn and trust the teams and the systems they are creating.</li><li>Investors: They may need to know many details about your systems to get on board.</li></ul><p>There is obviously a certain tension between the need for secrecy and the need to share information. This article explores the various things that could be shared and argues for creating strong norms around information sharing. An in depth discussion of openness can be found in <a href="https://nickbostrom.com/papers/openness.pdf">Strategic Implications in Openness in AI</a>.</p><p>Instead of radical openness or complete secrecy, the idea of translucency might help navigate the tensions.</p><h4>Translucency</h4><p>There are many different things that organisations can share. Sharing all of them would make the organisation fully transparent. Sharing none of it would make it opaque. So by analogy, sharing parts of it would make it translucent. It is worth enumerating the different things that can be shared as it is easier to share classes of information than it is to share partial information.</p><ul><li>Road map: The high level goals the organisation is pursuing.</li><li>Project management information: The day to day tasks being carried out by the team. E.g. trello boards . This can be important information to make sure nothing is being worked on in secret (if all employees are accounted for).</li><li>Testing Regime: The black box testing regime the AI system has been put through. This would tell you something about both it’s capabilities and how rigorous the testing was.</li><li>Architectural Diagrams and <a href="http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions">Architectural Decision Records</a>: This can give you an idea of how it is supposed to work, without giving you complete code level access.</li><li>Source Code: Generally considered the crown jewels for a software project. This would allow people to continue the work with minimal problem.</li><li>Build artifacts: Compiled software. While it does give the person shared with some benefits and it could be reverse engineered to get the full source code. But it does put a bit of speed bump in that process.</li><li>Tooling: The miscellaneous bits of software used by software programs that aren’t their core product.</li><li>Translucency Relationships with other organisations: As well as information about their AGI products, knowing there is sufficient oversight of the teams is important for trust from the general public.</li></ul><p>Teams would generally share information with other teams on a case by case basis increasing the level of cooperation as they got more comfortable. Teams would probably only cooperate if they had roughly comparable goals and metrics for what a good outcome looks like.</p><p>There are somethings I think should be shared in general with the public. The road map of work should be public to give people time to think about their jobs. Also the testing regime should be released well before the product is released to give people time to criticise it, especially if it is in a safety critical situation (self-driving car, robot surgeon, etc).</p><h4>Norm enforcement</h4><p>If you share information with some one else, you may not want them to share information with bad actors or your competitors. How do you make sure that people don’t casually share information with other people? As a company you could maybe sue the people that breached your trust, but it might also be a good plan to have agreements with other people you share information with such that if someone betrays you by sharing your information, you and they will also stop sharing information with them.</p><p>You can have similar agreements to stop sharing information, if they fail to share crucial information with the public or regulators (or they only share part of it from you, when they claim to share it all).</p><p>It would probably be more stable if there was explicit information sharing contracts, rather than just norms. As there would be punitive damages other than just loss of future sharing. There might be some value in having standard contracts written up that people can just adopt.</p><p>If we can build a large community that works together in this way, then they might be able to progress quickly and more safely than separately competing groups.</p><h4>State Actors</h4><p>It is hard to stop state actors taking your information, if they are determined. In the case where there is an ongoing infosec arms race that may escalate, in order to avoid your technology being part of the escalation, you may need to delay development of potentially dangerous AGI until there is international agreements that might bind the nation states. Or a cease fire.</p><h4>Conclusion</h4><p>Information sharing around AGI is hard. Done right it, it might be a force to create incentives for AI development beneficial to all. Done badly it might lead to arms races and bad actors being able to exploit the unprepared public.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=206db4627bcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breaking the Competent/Disempowered User dichotomy.]]></title>
            <link>https://medium.com/@eb4890/breaking-the-competent-disempowered-user-dichotomy-abe3049bee97?source=rss-14885c3a9830------2</link>
            <guid isPermaLink="false">https://medium.com/p/abe3049bee97</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Will Pearson]]></dc:creator>
            <pubDate>Tue, 06 Mar 2018 21:57:52 GMT</pubDate>
            <atom:updated>2018-03-06T21:57:52.410Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>One of the goals of off-piste architecture is to reduce what I think of as the Architecture/ User friction. This friction is caused by a difference between what the architecture expects of the user of a computer system and what the user is capable of. There are two different views that we have currently. The first and oldest is that the user is competent at administering their computer. The latecomer view is that the users are not competent to manage their computer and must be prevented from having any impact on how the lowest levels of their system operates, for their own protection.</p><p>Neither view is right. This causes friction between the user and the computer. Either the user makes changes they shouldn’t do or they cannot have any influence into the system as a whole even when it safe to make a change.</p><p>The computer architecture is not what you normally change when you are trying to make computers easier to use. You normally change higher level components, for example by improving the layout of the user interface or by giving context sensitive help. You don’t go monkeying around with the architecture and operating system.</p><p>But if we want to make computing significantly better for people we should change as much as we need to. The deeper the change to how we think about computing, the bigger the potential pay off. The greater the cost as well.</p><p>However this friction is being woven into the foundations of our society as we embed computers into more and more things. It makes sense to get this right and get it right now. It will only get harder to change.</p><h3>PEBUAI</h3><p>There is an unkind phrase in IT, <a href="https://en.wikipedia.org/wiki/User_error#PEBKAC">PEBKAC</a>, it stands for problem exists between keyboard and computer. It is used to indicate a user error, the user not understanding the system enough to use it properly.</p><p>However with more and more computers being deployed around us and the software we are using becoming ever more complicated, we cannot be expected to be the expert users of all of our all of their systems.</p><p>Is it a PEBKAC when you don’t patch your system (or do patch your system with <a href="https://www.theverge.com/2018/1/9/16867068/microsoft-meltdown-spectre-security-updates-amd-pcs-issues">a bad patch</a>), when you don’t install your antivirus or when you are tricked by a very good phishing website? Should these things actually be a prerequisite for interacting with the internet? At some point we should no longer blame the relatively limited humans but instead put the blame the ever more computationally powerful computers. The problem exists between the user and the internet. That is the problem is in the hardware and software that make up all of our machines, it does not meet our needs.</p><p>One possible way of mitigating some of these problems is offloading the maintenance of our systems to the people we buy the systems from. However we also cannot rely on manufacturers maintaining security or supporting servers. If the maintenance of our internet enabled device is no longer a profitable business, it won’t be maintained.</p><p>Humans and computers have not always had such friction, at the birth of modern computing things were relatively harmonious. However the model of interaction that was settled on then has set us down our current path.</p><h3>Dawn of modern computing</h3><p>In the era of punch cards the admin/computer relationship made sense. Expert programmers had simple programs that they understood, that they wanted to run. There were no viruses, no complex software stacks and no other operating system to interact with. Things were pretty simple back then, pre-time sharing and file storage, there was not much you could muck up. The user could emulate what was going on in the whole system in their head, just a lot slower. Users were in control.</p><p>We, to this day, still expect users to be either this competent or thoroughly incompetent to the point where they have no influence into the low level operation of the system. It is a binary decision.</p><p>One example of this competent/disempowered dichotomy is in how a computer first boots up.</p><h3>Giving it the Boot</h3><p>What happens when you boot a computer is that after the BIOS runs, in simple terms the BIOS picks a defined point on your storage to load a program from. That program is the boot loader.</p><p>The boot loader’s heavy responsibility is to load the rest of the operating system. It could chose to infect every program it loads with malware or just load plain malware.</p><p>This boot loading step happens in computers, phones, thermostats etc. Do users have the required skill or knowledge to be able to maintain all those things?</p><p>One alternative, introduced by UEFI, is to have<a href="https://insights.ubuntu.com/2017/08/11/how-to-sign-things-for-secure-boot"> code signing all the way to the operating system</a>. But that means the user cannot change things at all, unless it is by loading code signed by someone else or them figuring out how to sign things yourself.</p><p>The user may want change things about the boot loader and or the operating system it loads. Perhaps you want to speed it up or remove a feature that has been added.</p><p>But if you are given control you do not have the knowledge to use it wisely.</p><h3>Breaking the Dichotomy</h3><p>So what is the alternative? Rather than having a single point of failure or control, our computer architectures could automatically load multiple redundant programs that interact to form a functional computer.</p><p>The user could then load a new component and if it failed or proved to be malicious, another good component could substitute for it. No component would have overall control, there would be no critical lynch pin to fail.</p><p>I think that Agoric computing has a place in this view of computing, bit it might not. If we want to get better computers, people working in technology need to be questioning how our systems work from the ground up. <a href="https://offpistearchitecture.wordpress.com/">Off-piste architecture</a> is the beginning of that journey.</p><p>Without going on that journey we will be embedding two wrong views of users in the computational substrate of our civilisation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abe3049bee97" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>