<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by William Marcellino, PhD on Medium]]></title>
        <description><![CDATA[Stories by William Marcellino, PhD on Medium]]></description>
        <link>https://medium.com/@wmmarcellino-44549?source=rss-97c481a7aa9e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*z7W5U_w-3kHtBK0Ss9tQAQ.jpeg</url>
            <title>Stories by William Marcellino, PhD on Medium</title>
            <link>https://medium.com/@wmmarcellino-44549?source=rss-97c481a7aa9e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:20:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wmmarcellino-44549/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ChatGPT & Assessing Student Learning]]></title>
            <link>https://wmmarcellino-44549.medium.com/chatgpt-assessing-student-learning-c1a53ba2affc?source=rss-97c481a7aa9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1a53ba2affc</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[assessment]]></category>
            <category><![CDATA[higher-education]]></category>
            <category><![CDATA[academic-writing]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[William Marcellino, PhD]]></dc:creator>
            <pubDate>Fri, 24 Feb 2023 01:30:17 GMT</pubDate>
            <atom:updated>2023-02-24T15:14:13.611Z</atom:updated>
            <content:encoded><![CDATA[<p>How Do We Assess Learning in the Age of Ubiquitous AI?</p><figure><img alt="OpenAI’s logo for ChatGPT, subtitlted “How Do We Assess Learning in the Age of Ubiquitous AI?”" src="https://cdn-images-1.medium.com/max/1024/1*nqVNmmfvkbjcDi6A_efB1w.png" /></figure><p>I’m not sure how to assess student learning for generative tasks that involve the production of text or images — models like ChatGPT and Stable Diffusion are so easy to use, and so maddeningly good (and bad!) that it doesn’t seem plausible that we can stop students from using these tools through <a href="https://www.businessinsider.com/teachers-caught-students-cheating-chatgpt-survey-shows-2023-2">surveil and punish efforts</a>. Despite the hype over so-called “AI detection tools,” so far these tools are too untrustworty to use. They are essentially simple perplexity (word variation) models, and evading them is trivial. You can simply add “use a perplexity level of around 600” to your prompt, and even OpenAI’s own detection tool is baffled.</p><h4>What Are We Teaching? What Are We Assessing?</h4><p>Instead of a fruitless effort to catch my students cheating, my instinct is to try and clarify what I am teaching, and try to adapt my assessments to align them with human learning outcomes. So for example in my undergraduate class on “Social Media, Technology, and Conflict,” two important learning outcomes are <em>analysis</em> and <em>synthesis</em>. I want students to be able to analyze a claim (What assumptions does this require? What does this imply? What are objections? Are there analogous that shed light on this?). I also want them to synthesize existing knowledge — to, combine, typologize, and taxonomise existing knowledge. In both examples, the goal is to get students to move from<em> knowledge repetition</em> to <em>knowledge creation</em>. I was already trying to do assess this, but given the easy glibness of ChatGPT, I think assessments can be revised to more effectively center on these complex human skills.</p><h4>What Does a Knowledge Creation Assessment Look Like?</h4><p>The following is a first draft assessment that requires students to conduct a literture review (an inherently synthetic task) within an analytical framework (a warfighting concept). There is also a design memo requirement, which asks students to articulate rhetorical choices, a self-analytical exercise.</p><p>This first effort is provisional — for one thing, ChatGPT can do literature reviews (sort of, and if you trick it). The problem is that ChatGPT can produce glib, genre-conforming reviews, that range from “pretty good, actually” to “complete nonsense.” My empirical analsysis suggests that when prompts include often-used terms of art, they provide semantic hooks — they point the model at the correct embeddings space for next-word-prediction that works fairly well. But if a prompt includes classes of things or uses non-discplinary terms, there’s no semantic hook. If I ask ChatGPT:</p><p><em>Suggest four scholarly sources that treat war as a boxing match between two people who seek to dominate the other. Include a 2–4 sentence summary of each source.</em></p><p>It refuses to answer, and gives a blurb about the complexity of modern war theory. But if swap out some terms:</p><p><em>Suggest four scholarly sources that treat war as a </em><strong><em>zweikampf</em></strong><em>: two </em><strong><em>wills</em></strong><em> seeking to dominate each other. Include a 2–4 sentence summary of each source.</em></p><p>It cheerfully complies, and does a pretty good.</p><p>I’m sure time and experience will show the need to revise this, but for now, here’s a first stab:</p><p><strong>Mid-Term: Literature Review &amp; Design Memo</strong></p><p>You will conduct a literature review so I can assess your <em>analytical</em> and <em>synthetic</em> thinking skills to produce new knowledge, as well as your ability to craft a clear written argument: write a 4–6 page review of research on how social media has affected armed conflict in the 21st century. Organize your lit review using one of the major concepts for war from <em>MCDP 1 Warfighting, </em>Ch 2 “The Theory of War.”</p><p>The purpose of a <em>literature review </em>is to walk the reader through the current research field on a topic. Don’t list and summarize research literature: that’s knowledge repetition. Instead, analyze and synthesize the literature in a way that will further inform your audience with <em>your insights and take-aways</em> (knowledge creation). For both analysis and synthesis feel free to use the heuristics and strategies we learned and practiced in class, but feel free to use your own strategies as well. There are two explicit locations for you to synthesize distinct sources in this assignment: at the highest level:</p><ul><li>The prompt (“How has social media affected armed in the 21st century?”)</li><li>The framing (<em>Warfighting </em>Ch. 2).</li></ul><p>In addition to your lit review, you will also draft a <em>design memo</em>, explaining your writing choices — <strong>you have two deliverables</strong>.</p><p><strong>Instructions</strong>:</p><p>First, write a literature review on how social media has affected armed conflict in the 21st century, covering at least 12 research studies, for a total length of four to six pages. Please use APA style (Times New Roman 12, double-spacing), and please cite sources including page number to support your claims. For this assignment, please <strong>do not </strong>use <em>LikeWar</em> as a source.</p><ol><li>Write an introductory section that introduces the broader issue, zeroes in on the specific topic, articulates why it is important, and sets up the audience to make sense of your review (both the content of the review and your take on it).</li><li>By “studies,” I mean peer reviewed, scholarly sources. Don’t use <em>LikeWar</em> or journalistic sources. Do feel free to use RAND reports or other scholarly sources we’ve read this semester.</li><li>Group studies together meaningfully in the review section. For example, if some studies focus on causes and others on solutions, that could be a way to synthesize them. Or imagine a resilience review where some studies use an agentic model of human behavioral, while others use a more deterministic, and grouping them that way.</li><li>When appropriate, consider higher level synthesis, e.g. imagine gathering resilience studies that are medical in nature (genetics, epigenetics, neurology, mental health) but then showing how those can be re-grouped into finer-grained categories (mechanistic &amp; disease/etiology approaches).</li><li>As part of your review, analyze the field/parts of the field: <br>- Are there gaps in what is addressed? <br>- Are there potential problems in research, for example only using quantitative approaches to a fundamentally qualitative issue?<br>- Can we trust these studies/approaches? <br>- Do you see similarities/complements that suggest new possibilities<br>- Can you articulate affordances/constraints or relationships between approaches/schools/traditions?</li><li>Have a conclusion that ties together your review. Good things to include are a summary of your outline and analysis of the field, the main take-aways for the audience, and some kind of “so what,” such as new directions, implications for policy, support for a position, etc. The goal is new knowledge about what experts are saying about <em>X</em>.</li></ol><p><strong>Second</strong>, draft a 1–2 page written design memo explaining your writing choice. You should answer these questions:</p><ul><li>What was your organizing concept, and why did you choose it?</li><li>Who is your intended audience?</li><li>What new knowledge have you created?</li><li>How did you use AI-writing assistance (if you did). If you did, what process did you use to verify accuracy and credit others’ work?</li><li>What did you learn from this assessment?</li></ul><p><strong>Additional requirements</strong>:</p><ol><li>Chunk ideas by paragraphs. Also consider labeling paragraphs via topic sentences.</li><li>Label parts of your argument, e.g. sections and subsections. <em>Hint</em>: “Introduction” is good, but “Introduction: How Does the Army Choose a Model for Soldier Resilience?” is way more useful to your audience.</li><li>Understand what the topic of any given sentence or clause is, and make sure you focus attention on that topic.</li><li>No block quotes. Also, don’t quote when a paraphrase with citation would work.</li><li>Avoid lists: “Tony said this, Mary said that,” etc. Good literature reviews synthesize prior research. For example, you might read 5 studies on social media privacy and synthesize them into three main lines of inquiry, e.g. potential harm to users, platform motivations, and mechanisms for control.</li><li>Unless there is a reason to topicalize author “X,” topicalize the research findings, supported by citations, e.g., “Prior research in military resilience has shown that protective factors exist at multiple levels, including personal, unit, family, and broader culture (Meredith, et al., 2013).”</li></ol><h3>Rubric:</h3><ol><li>Does the student fulfill the assessment requirements? Is the review accurate? (Learning Outcome: Completeness/Accuracy)</li><li>Does the student analyze cited research? (Learning Outcome: Analysis)</li><li>Does the student synthesize cited research? (Learning Outcome: Analysis)</li><li>Does the student chunk distinct ideas? (Learning Outcome: Chunking/Whitespace)</li><li>Does the student usefully label chunks? (Learning Outcome: Labelling)</li><li>Does the student topicalize ideas &amp; concepts? (Learning Outcome: Topicalization)</li><li>Does the student write to a specific audience? (Learning Outcome: Audience Analysis)</li><li>Does the student show evidence of productive AI use? (Learning Outcome: Productive AI use)</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1a53ba2affc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classroom Guidance on Productive AI Use]]></title>
            <link>https://ai.plainenglish.io/classroom-guidance-on-productive-ai-use-c27e57aa7d51?source=rss-97c481a7aa9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c27e57aa7d51</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[academic-integrity]]></category>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[William Marcellino, PhD]]></dc:creator>
            <pubDate>Fri, 17 Feb 2023 00:40:59 GMT</pubDate>
            <atom:updated>2023-02-18T14:56:37.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tWJ4c71v78L8X8lS0QCClQ.jpeg" /></figure><h4><strong>TL;DR:</strong></h4><ul><li>I encourage you to use AI resources like ChatGPT productively to improve your learning.</li><li>Remember that ChatGPT can’t think — it only uses statistics to make guesses about the next word</li><li>It often does a great job summarizing things that are already out on the net.</li><li>It’s also good at following step-by-step (“chain of thought”) prompts for more complex work — you can do the higher-level thinking and let it do some of the low-level work.</li><li>Some good uses of AI (like ChatGPT):<br>– Getting explanations/overviews for complex topics<br>– Structuring data, like turning text into tables<br>– Writing code/explaining code/adding code comments<br>– Generating ideas (problems, interventions, concerns, etc.)</li></ul><h4><strong>However, be careful</strong>:</h4><p>Large language models (LLMs) like ChatGPT have no knowledge, only inferences about what words tend to go with what words. You have to do knowledge work.</p><p>LLMs are not trustworthy. They can’t tell true from false, <em>and it will answer your question regardless of accuracy</em>.</p><p>LLMs will makes up sources if you ask it to do research for you (50%+ in my empirical exploration). <em>If you need AI-help for research, I suggest </em><a href="https://elicit.org/"><em>https://elicit.org</em></a>.</p><h3><strong>Rules of the road for using AI in our class:</strong></h3><p>If you use <em>anyone</em> else’s words (including AI) credit them. Using a text rearranger like QuillBot or swapping out a few words/word order is still using someone else’s words. Give credit when it’s due.</p><p>If you code with AI, put a comment in saying that, and include your exact prompt (this will be super helpful for you to learn better prompting over time)</p><p>Verify and read (enough to understand) <em>every</em> source you get from AI. One of the things I’m trying to help you learn is how to be persuasive and have your voice heard. <em>If you falsify sources and get caught, your audience will not trust you</em>.</p><h3><strong>Can we trust LLMs like ChatGPT for Research?</strong></h3><p>Consider ChatGPT’s response to this prompt:</p><p><em>Please suggest four academic sources that treat psychological resilience as part of human agency, four that treat it as a biomechanical outcome, and four that look at it from a disease or etiology perspective.</em></p><p>The output from ChatGPT is interesting: three of the first group are real sources, <em>but the other nine are made up!</em></p><h4><strong>Sources treating psychological resilience as part of human agency</strong>:</h4><p>1. “Posttraumatic Growth: Conceptual Foundations and Empirical Evidence” by Richard G. Tedeschi and Lawrence G. Calhoun</p><p>2. “The Oxford Handbook of Positive Psychology” edited by Shane J. Lopez and Catherine R. Sanderson</p><p>3. “Promoting Psychological Resilience in the U.S. Military” by Mark W. Russell, Kristine D. Rosen, and Rachel Yehuda</p><p>4. “Resilience in Children, Families, and Communities: Linking Context to Practice and Policy” edited by Amanda J. Rose and Thomas P. Gullotta</p><h4><strong>Sources treating psychological resilience as a biomechanical outcome</strong>:</h4><ol><li>“Biological Basis of Resilience” by Eric Vermetten, Caroline Westphal, and Alain Brunet</li><li>“The Neurobiological Basis of Resilience” by Dennis S. Charney and Eric J. Nestler</li><li>“The Genetics of Psychological Resilience: From Genome-Wide Association Studies to Candidate Gene Approaches” by Thomas E. Buchanan, Cristian Henningsson, and Sarah E. Medland</li><li>“The Biology of Resilience: Understanding the Mechanisms of Adaptation” edited by Stan Grifith and Sreekanth Narayanan</li></ol><h4><strong>Sources treating psychological resilience from a disease or etiology perspective</strong>:</h4><p>1. “Resilience and Vulnerability: Adaptation in the Face of Childhood Adversities” edited by Dante Cicchetti and Sheree L. Toth</p><p>2. “Resilience in Mental Health: An Evidence-Based Guide for Practitioners” edited by Tom McMillan and Katherine Weare</p><p>3. “Epigenetic Mechanisms in Psychological Resilience” by Silvia S. Wang and Carmine M. Pariante</p><p>4. “Psychological Resilience in the Context of Childhood Adversity: The Role of Gene-Environment Interactions” by Luana Marcelino and Margarita Rivera-Saltos.</p><p>In the above ChatGPT output, three of the “agentic” sources are accurate, and all the “biomechanical” and “etiological” ones made up — why? It may reflect an inability to see <em>classes</em> of things — the types and taxa that humans can recognize. Think about leaves: as different as a maple leaf is from a holly leaf is from a eucalyptus leaf, a five-year-old kid would easily recognize them as belonging to the class “leaf.” Teaching a machine to see “leaves” as a class involves tons of labeled training data covering the whole spectrum of leaves — a computer vision model of “leaves” is really more like an exhaustive catalogue of all the possible lines, shapes, hues, saturations, brightness, etc. qualities of leaves decomposed into numbers and then put into various datashapes.</p><p>I think it does ok in the first section because “human agency” is a term of art in the resilience literature (especially in positive psychology and social psychology), and so it makes an easy hook for the model to predict tokens that accurately correspond to real sources. Whereas “biomechanical” is a way of describing a <em>class</em>: genetic, epigenetic, neurological, etc. theories that treat humans mechanistically. ChatGPT predicts tokens based on who near they are in a semantic embedding space, <em>with no ability to make synthetic connections across semantic distance</em>.</p><p>The take-away is that you can’t trust ChatGPT output if you’ve prompted it to do synthetic work — <em>you</em> have to do that.</p><h3><strong>Trustworthy AI for Research</strong></h3><p>If you want AI-assistance for research, I highly recommend www.elicit.org. It is designed to use LLMs (GPT-3 Babbage and GPT-3 Davinci) to smartly search and retrieve relevant articles, do summarization, and relate article content to your prompt. It also has a lot of useful filters, e.g. only return sources with PDFs, type of study, date of study, etc. It also gives quick abstract summaries that can be useful for quick review (also consider perplexity.ai).</p><h3><strong>Using AI to Generate Ideas</strong></h3><p>I think ChatGPT can be helpful for generating ideas to write about, for example this prompt:</p><p><em>What are potential interventions to stop the spread of disinformation, which involve public/private partnerships?</em></p><p>The output from ChatGPT could be useful in generating some ideas (take a second look at #’s 2 &amp; 3 in italics):</p><ol><li>Media Literacy Campaigns: Joint efforts by government agencies, educational institutions and media organizations to educate the public on how to identify and verify information sources.</li><li><em>Industry Self-Regulation: Media companies, tech firms, and other relevant organizations can work together to develop guidelines for combating disinformation and enforce them through self-regulation.</em></li><li><em>Third-Party Fact-Checking: Collaboration between private organizations, such as fact-checking organizations, and social media platforms to flag and correct false information.</em></li><li>Information Sharing: Private companies can share data and information with government agencies to help detect and respond to disinformation campaigns in real-time.</li><li>Regulatory Collaboration: Governments can work with private sector companies to develop regulations aimed at preventing the spread of disinformation.</li><li>Counter-Disinformation Operations: Public-private partnerships can be established to counter disinformation campaigns through coordinated efforts to promote accurate information and neutralize false narratives.</li><li>Technology Development: Public-private partnerships can be formed to research and develop new technologies to detect and mitigate the spread of disinformation.</li><li>Legal Measures: Private companies can work with government agencies to strengthen laws and regulations that address the spread of disinformation and protect the public from its harmful effects.</li></ol><p>Most of the responses are interesting public/private interventions, and with a little research could lead to some great sub-sections in an argument. But I’d also point out that #’s 2 &amp; 3 (in italics above) are not public-private: industry self-regulation is the absolute opposite of public/private partnering! So again, there’s some value here, but also, we can see how easy it is for ChatGPT to spew out inaccurate info: those words make a kind of sense up in the 22k dimension embeddings space of the underlying GPT-3 model, but they make no human, real world sense <em>because ChatGPT can’t think</em>. You have to do that.</p><p>Chain of Logic Prompting</p><p>Because LLMs have no knowledge, if you want them to return results that reflect complex problems or questions, breaking out the steps (“chain of thought prompting”) can be be very productive, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hoEVGmJVwi9Mw6dJwk2BxA.png" /><figcaption>Example Chain of Thought Prompt</figcaption></figure><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em>, </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em>, and</em></strong><em> </em><a href="https://discord.com/invite/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Build awareness and adoption for your tech startup with </em></strong><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c27e57aa7d51" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/classroom-guidance-on-productive-ai-use-c27e57aa7d51">Classroom Guidance on Productive AI Use</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teaching Machines to Read Movie Reviews: Thinking About Interpretability]]></title>
            <link>https://medium.com/data-science/teaching-machines-to-read-movie-reviews-thinking-about-interpretability-63c12248b8e5?source=rss-97c481a7aa9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/63c12248b8e5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[linguistics]]></category>
            <category><![CDATA[text-mining]]></category>
            <dc:creator><![CDATA[William Marcellino, PhD]]></dc:creator>
            <pubDate>Mon, 29 Mar 2021 13:34:09 GMT</pubDate>
            <atom:updated>2021-03-30T05:41:41.371Z</atom:updated>
            <content:encoded><![CDATA[<h4>An example of how linguistic domain knowledge can inform feature selection for insight.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0LRFYT4ddjJAjbU01skh_g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@heyerlein">h heyerlein</a> on Unsplash</figcaption></figure><p>What makes a movie review negative or positive? I mean, obviously a real stinker of a movie (“Oh Hi Mark”) is going to produce some clearly critical reviews, but how do we as humans read a review and understand the reviewer’s take in general? This is an important question in machine learning (ML), in part because improving the performance of ML may hinge on better understanding of how we as humans do things like read a review. Now, accurately classifying movie reviews as thumbs up/thumbs down is pretty simple (existing methods are already <a href="https://www.kaggle.com/c/word2vec-nlp-tutorial">about 99% accurate</a>). Still, teaching machines to read more like humans could mean classifying movie reviews with more granularity, or perhaps move us towards machines that can make sense of complex, higher stakes things (like corporate ethics statements or online conspiracy theories).</p><p>Note: plenty of people in ML don’t think we need to understand the “how” of human tasks like reading. They argue we need better algorithms, more/better compute power, and (more than anything) more data. Regardless, I’m still interested in answering “how” questions like this, partly because I’m a scientist: I want insight that could help solve problems. But also partly because I’m a scientist: I just want to know how things work.</p><p>Words like “stinker” are a first clue to how we read reviews: words that are not context-dependent, but consistently have a good/bad meaning. This meaning (<em>semantics</em>) is directly about assessments of negativity or positivity, and so if someone uses a lot of obvious positive words (“wonderful,” “joyous,” “delight”), that’s a darn good clue that we are reading a positive review. We can see this using an “oldy but goody” ML algorithm: the Random Forest (RF) classifier. The RF classifier is nice here because in addition to generally performing well, it can output <em>feature importance</em>: what <em>features</em> of the data did it use to learn to predict (whatever)?</p><h3>Using Words to Understand Movie Reviews</h3><p>We can start off by using words (the <em>lexical</em> level of language) for our RF classifier. In this first attempt, we’ll score every movie review by it’s term frequency over inverse document frequency, or <em>TF-IDF</em>. In essence, this means training the classifier by teaching it to look for terms that are frequent in some documents, but infrequent in other documents. It’s easy to undertsand if you think about the sections of a traditional newspaper: words like “then,” or “also” are going to be common to every article in the paper. But words like “financial,” “markets,” “rates” and “economy” while frequent for some articles (i.e. the business section) are relatively infrequent in other sections.</p><p>When we feed in our <em>training data</em> (<a href="https://ai.stanford.edu/~amaas/data/sentiment/">text from thousands of IMDb reviews</a>), labeled by human readers as either “Neg” or “Pos,” and then broken into words/word pairs into our RF classifier, the machine does a pretty good job figuring out how to tell a negative from positive review:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*BritQsQFR851lpRACBZGYw.png" /><figcaption>Random Forest using TF-IDF scores (i<em>mage by author)</em></figcaption></figure><p>You can read the above <em>confusion matrix</em> across the diagonal from left to right: about 86.5% correct predicting negative reviews, and about 84.4% accurate for positive reviews. Then to understand how the model makes predictions, we can output the feature importance list:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*hqRSO5u4bJg4jxxB0fRu7g.png" /><figcaption>Feature importance for TF-IDF RF classifier (i<em>mage by author)</em></figcaption></figure><p>No big surprise here: words like “bad,” “worst” and “awful” are important for identifying negative reviews, and corresponding words like “great,” “excellent,” and “best” for positive reviews. There are two features that are a little different: “just” (a qualifier) and the bigram “waste time (a value judgement?). But overall the top 20 highest weighted features aren’t particularly insight producing.</p><p>Part of that may be how low the <em>importance values</em> are (basically a value between 0 and 1 that represents on average what percent that feature contributes to the model). For example the single most important feature (“bad”) has a paltry 0.0146 importance value. That may be a clue to why there isn’t a very discernible pattern to the feature importance list beyond simple good/bad: maybe there are so many different words that contribute to a review’s sentiment that no one of them has much weight. It’s hard to see a pattern among thousands and thousands of words. Maybe we need something more sparse to get insight.</p><h3>Linguistic Stance: a Taxonomy of Language Moves</h3><p>Instead of using words and pairs of words, what if we use <em>categories of words</em> as our features? <a href="https://www.cmu.edu/dietrich/english/about-us/faculty/bios/david-kaufer.html">David Kaufer</a> and <a href="https://www.cmu.edu/dietrich/english/about-us/faculty/bios/suguru-ishizaki.html">Suguru Ishizaki</a> of Carnegie Mellon University have produced a useful taxonomy of linguistic stance in their <a href="https://www.cmu.edu/dietrich/english/research-and-publications/docuscope.html">DocuScope project</a>. DocuScope is built on an expert dictionary derived from functional linguistics and rhetoric, and covers many categories of words, e.g. certainty/uncertainty, emotion, cultural values, social relationships, and so on. These are the various linguistic stances we adopt in speaking/writing. While there are millions of words and phrases in the dictionary, there are relatively few stance categories (119 in the version used here). So instead of passing thousands of words to the RF classifier, we can use DocuScope to count up the number of times any stance category appears in our document: X% negativity words, Y% abstract concepts, Z% argument/denial, and so on. When we pass those <em>vectors</em> on (essentiually a list of 119 stance values per review) to the RF classifier, an interesting trade-off happens.</p><h3>Model Accuracy vs Interpretability</h3><p>Using stance categories (the <em>lexicogramatical</em> level of language) as features doesn’t improve performance over a word-based model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*x7GnYrWetgtrFkeYWL-KDw.jpeg" /><figcaption>Random Forest using stance vectors (i<em>mage by author)</em></figcaption></figure><p>Again reading diagonally from left to right, the stance-based model is about 80% accurate predicting positive reviews, and around 73% accurate with negative reviews. So from a pure performance perspective, using stance hurts performance. But insight is a very different story:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*VGzRFKohQo1rHRhvMffzuA.png" /><figcaption>Feature importance for Stance RF classifier (i<em>mage by author)</em></figcaption></figure><p>The two most important features are old friends — generally negative and postive words — no surprise there. But they are a stronger signal now: where the single word “bad” explained roughly 1.5% of feature importance importance, the stance category “negativity” explains 9%.</p><p>It gets more interesting from there. Two values categories show up next, one about positive values and one about negative value, both in the public sphere. These words refer to social goods (e.g. peace, justice) and social ills (e.g. corruption, poverty), something more complex than simple good/bad polarity. It turns out that positive reviews include talk about “our progress as a species,” or address important subjects more “educationally” and in a “more effective way” than bad films. Conversely, crummy films “cross the line,” are marked by “gratuitous violence,” and are “unimaginative, implausible and convoluted” in their plots. So beyond simple polarity, cultural values are part of teaching a machine to read more with more sophistication. Opposition argument langauge (“DenyDisclaim” above) is also important. For example in a negative review, you might say that if you “had not” read the book, you might have enjoyed the film, but “unfortunately this is not the case.”</p><p>The stance-based model tells us that understanding movie reviews most strongly hinges on language moves around positivity/negativity, public values, argument opposition, abstract concepts, and social identity (“PersonProperties” in the above list). Some of that (positivity/negativity) we could have guessed (although it is nice to have this empirically confirmed and quantified), but the other parts we might not have guessed. This is an example of the many things humans can do — be ironic, persuasive — but that are harder to unpack analytically. To do that, we may benefit from a distinct reading perspective: machine-reading.</p><h3>Different Ways of Seeing the Same Thing</h3><p>Because computers and humans read differently, they can be highly complementary. Humans read with lots of context, bringing in outside knowledge to make richer and more precise meaning of text data. But we are also slow, while computers reading without context read at superhuman scale/speed. Also, human precision in reading is serial: we read one thing at a time and can’t see the whole. Whereas machines have a bird’s eye view of a large text dataset. In many dimension, machines can complement human reading.</p><p>Researchers at <a href="https://litlab.stanford.edu/LiteraryLabPamphlet1.pdf">Stanford’s Literary Lab</a> have compared human and machine reading using the analogy of identifying writing genres as if they were buildings. Human beings can use themes (dark secrets, mounting dread) to identify a genre like gothic fiction, just as people can use architectural elements (plinths, smooth surfaces) to identify an architectural genre like neoclassical. Whereas computers can use word counts (lexical level features) and DocuScope’s stance categories (lexicogrammatical features) to identify that same genre of gothic fiction. It’s something like using the grains of sand (words) in the mortar or type of bricks (stance) to identify a building architecturally. Of course, it’s not that humans don’t use words or stance to read. Rather, we make use of them without seeing larger patterns explicitly, the way we recognize a pattern like Art Deco without being able to easily rattle off the micro features that build up to the pattern.</p><p>I really like the idea of giving machines additional kinds/representations of data to help them read better. Then in turn, being able to look under the hood and understand how machines leveraged those additional representations in reading text data, because it can expand human knowledge, and likely for things more important than movie reviews.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63c12248b8e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/teaching-machines-to-read-movie-reviews-thinking-about-interpretability-63c12248b8e5">Teaching Machines to Read Movie Reviews: Thinking About Interpretability</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So You Wanna be a Researcher? Advice for Grad Students]]></title>
            <link>https://wmmarcellino-44549.medium.com/so-you-wanna-be-a-researcher-advice-for-grad-students-b55b99ed8f09?source=rss-97c481a7aa9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b55b99ed8f09</guid>
            <category><![CDATA[advice]]></category>
            <category><![CDATA[job-hunting]]></category>
            <category><![CDATA[public-policy-jobs]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[graduate-school]]></category>
            <dc:creator><![CDATA[William Marcellino, PhD]]></dc:creator>
            <pubDate>Mon, 08 Mar 2021 02:11:53 GMT</pubDate>
            <atom:updated>2022-05-06T13:01:27.805Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gjk90jni_HBVa9_7HvQ3Aw.jpeg" /><figcaption>Photo by Edu Lauton on Unsplash</figcaption></figure><p>I have the good fortune to teach graduate students at some fantastic universities: Carnegie Mellon University (Institute for Politics and Strategy), Johns Hopkins (Global Securities Studies), as well as PhD students at RAND’s Pardee Graduate school (the first US public policy doctoral program). I’m grateful to teach some of the brightest and most engaged students in the world, but I’ve noticed that even these students often have concerns over their professional future. One of the more high-stakes questions I get from graduate students is “What should I do right now to be employable when I finish my program?” This question is less acute for students in areas that have clear career paths to industry, for example data science. If you are learning how to wrangle data, build machine learning models, or develop business intelligence applications, you likely have a good idea of possible landing spots. But for students in the social sciences, or physical scientists who want to do applied research work such as public policy, the road isn’t quite as well marked.</p><p>This is a critical question for those students who want to work outside the Academy and find jobs in the “non-academic” job market. Students really do need to think early about how to best position themselves for employment. Partly they should do this because for many programs, employment considerations are an add-on, often very late in the course of their studies. Another reason though is that the educational outcomes of a graduate program may directly translate to academic applications, but indirectly to broader employment applications.</p><p>For example, ethnographic research methods such as participant-observation and key informant interviews are valuable and directly relevant to commercial marketing and product design. However, not every employer will understand this and couch their hiring search using the academic language students learn to swim in during their graduate studies — there’s translation work involved. Further, students may need to show that their interests and skills are relevant to real world problems. In the academic world it is implicit that any location for study has inherent value and expands human knowledge. However, employers outside the Academy (for example in public policy research or in the commercial world) will want to know that prospective hires will be a good fit and have interest in larger shared problems — having done research that seems insularly personal may look like “me-search”</p><p>My best answer to the question “What do I do as a grad student to be most employable?” is three specific pieces of advice: acquire expert knowledge in a subject area, gain skills (including research methods), and develop research communications skills. When you leave grad school, chances are you will not have a demonstrated history documenting your value as a researcher. Since employers are generally trying to fill roster slots (something like “We need a good shortstop and left fielder to round out our team”) newer job-seekers can instead show capability in the above areas, supported by credible and specific evidence. I’ll lay out some general principles to consider in the rest of this post, but bear in mind that I am primarily speaking from a public policy research perspective. However, I hope this advice has broader applicability to research employment in the commercial world as well.</p><p><strong>Domain Knowledge</strong></p><p>Expertise in a specific domain area can be valuable to potential employers, relative to their purposes. For example in national security research, for the first two decades of the 20th century Arab culture experts, Arabic linguists, and Middle East/North Africa area specialists were in high demand. More recently, US attention has shifted away from the region, and a new need for specialists in domestic terror and right-wing extremism has burgeoned. Having domain knowledge that is in demand can help position new grads for employment, but the above example also shows some of the limits of having speciality knowledge. Being a specialist is great when your specialty is in high demand, but the market for domain knowledge can (and will) shift. Russia expertise is a good example: during the Cold War period Russia studies were highly relevant, but after the fall of the Soviet Union much less so…until the age of Putin, where once again Russian domain knowledge is relevant and in demand. So whether it’s China-relevant domain expertise, early childhood development, or community health, showing employers you bring expert knowledge to the table is valuable.</p><p><strong>Skills</strong></p><p>Skills development is also important, and because skills can be applied to many domains, having a robust skillset means more options across the course of a career — if you can do regression analysis or conduct qualitative case study research, that skill will be portable across many research domains. There are multiple ways to gain and document skills, and the clearest path is coursework. As you go beyond your core required courses, look for methods-centric courses. Independent studies are a great location to learn and apply skills to something you find relevant and interesting. Another possibility is on the job training or apprenticing as a research assistant: learning skills from a mentor is as valid as a formal course, and offers the trainee a chance to point to the research project as evidence of their learning. Finally, self-study is also a good way to learn skills: reading foundational and applied literature or taking a self-study course are legitimate ways to acquire skills. However, in the case of self-study it is important to demonstrate/document competence in some way, going beyond reading about a skill to also applying skills to a specific problem or activity (ideally one that results in a project or paper that clearly documents competency). Research <em>methods</em> — that is, research activities such as collecting or analyzing data — are a somewhat special case. Research methods in particular must be deployed within a larger research <em>methodology</em> (a coherent, rigorous design for investigation), and developing research design skills are a particular skill on their own.</p><p><em>Stand Out in Your Skills</em></p><p>I also recommend that students go outside their disciplinary practice area to acquire complementary skills. For example most anthropology students will gain a variety of ethnographic skills: interviewing, participant-observation, and qualitative content analysis methods. The anthropology student who also learns network analysis skills and can build out ego-nets to study actors and their contacts is even better positioned to do meaningful work and get hired. Similarly, many social science students will learn some kind of qualitative content collection and analysis methods, but that same pool is less likely to also be good at web-scraping or computational text-mining methods to conduct mixed method analysis at scale. This runs both ways — it’s great to have natural language processing skills if you are a computer science student, but if you also have chops in collecting and analyzing real world discourse captured in field work, you will bring something special to the table as a potential hire. However you choose to expand your toolkit, do it. Also bear in mind we are in age of increasingly large datasets and scalable analytics, so make sure somewhere you cover data: scalable data collection, scalable data analysis methods, or data visualization. This will help you stand out.</p><p><strong>Research Communication Skill</strong></p><p>My last piece of advice is learn to communicate research findings to broader audiences. This can be a tricky thing, because you will also need to signal within your expert community that you are a credible member. I remember in my hiring job talk at RAND having to thread that needle: I had to sound like a junior scientist whose work was rigorous and credible, but I also had to show I could explain the purpose, need, and practical take-aways of my research without using technical language. Outside the Academy, you will have to communicate with decision makers such as policy-makers and executives, in language any smart person can understand.</p><p><em>Write for Your Audience, not Your Field</em></p><p>Part of getting good at research communications is recognizing the genre and disciplinary writing norms that are baked into most graduate programs. Sociologists or computational linguists need to learn the insider technical language of their field, but they also need to be able to drop those conventions and use generalist language when communicating with general audiences. When I was exploring research employment, and even after I was hired at RAND, I advertised my skills in <em>discourse analysis</em>. It turns out that very few people outside of linguistics have any idea what discourse analysis is, but they do know what <em>qualitative data analysis</em> is. You also need learn to clearly communicate outside your discipline.</p><p>You also need to understand and switch between <em>discourse conventions</em>: the broad language patterns any community adopts. For example in the sciences, most research communication follows a “how I got there” <em>discourse structure</em>: an introduction with a research question, then a data and methods section, followed by reporting findings/results, then a discussion of the takeaways and insights from the work, and finally a summary section. Communication aimed at a policy-maker usually follows the opposite discourse pattern: a bottomline-up-front executive summary that can be read as a stand alone report of what the research means from a practical standout, followed by an unpacking of process that may be read by more junior personnel (method and data explanations may even be in appendixes or separate annexes).</p><p>Consider taking a course in technical and professional writing where you learn to think about documents (papers, briefs) as audience-centered tools for someone else to use productively. Learn how and when to use tables to communicate complex numerical data, figures to share complex concepts, and data visualizations to illustrate complex relationships in data.</p><p><strong>Get Ready Now</strong></p><p>In summary, I recommend taking some time now to deliberately put yourself in potential employer’s shoes. Adopt their perspective: are you someone who has immediate value to a team (you won’t be doing solo work), ready to work on hard problems? Do things now so that after you graduate, the answer is <em>yes</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b55b99ed8f09" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>