<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Zelig-dev - Medium]]></title>
        <description><![CDATA[Documenting the development of the Zelig R statistical package - Medium]]></description>
        <link>https://medium.com/zelig-dev?source=rss----c6cfd6af8596---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Zelig-dev - Medium</title>
            <link>https://medium.com/zelig-dev?source=rss----c6cfd6af8596---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:32:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/zelig-dev" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Version 5.1.5, Beginning to Deprecate Time Series]]></title>
            <link>https://medium.com/zelig-dev/version-5-1-5-beginning-to-deprecate-time-series-5f76ce0dd025?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/5f76ce0dd025</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Sun, 19 Nov 2017 05:55:39 GMT</pubDate>
            <atom:updated>2017-11-19T05:55:39.219Z</atom:updated>
            <content:encoded><![CDATA[<p>We have been working for the past few months to make sure that Zelig is <strong>highly stable</strong> and <strong>clearly focused</strong> on bringing the most <strong>value</strong> to our users.</p><p>To achieve this goal, we recently decided to deprecate Zelig time series analysis (as called through models&quot;arima&quot;, &quot;ar&quot;, and &quot;ma&quot;).</p><p>A notice has been added in the 5.1.5 Zelig release. We plan to fully remove the current implementation of time series models in February 2018.</p><p>Time series models in Zelig have not progressed out of experimental status. They were notoriously difficult to maintain (see the many bug issues related to them) and most importantly, we have not identified a clear user story to focus development around. We never really answered the question: what quantities of interest were users most interested in finding?</p><p>Perhaps another implementation of time series models will be developed in the future when we have a better understanding of how Zelig could bring the most value to these models.</p><p>In the meantime, please also checkout the <a href="http://christophergandrud.github.io/dynsim/">dynsim</a> and <a href="https://github.com/christophergandrud/pltesim">pltesim</a> R packages which implement a simulation method similar to Zelig’s for specific quantities of interest.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f76ce0dd025" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/version-5-1-5-beginning-to-deprecate-time-series-5f76ce0dd025">Version 5.1.5, Beginning to Deprecate Time Series</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breaking into Zelig (5.1–2 released) and new team members]]></title>
            <link>https://medium.com/zelig-dev/breaking-into-zelig-5-1-2-released-and-new-team-members-d17738cf6ef1?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/d17738cf6ef1</guid>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Thu, 08 Jun 2017 16:55:36 GMT</pubDate>
            <atom:updated>2017-06-08T16:55:36.361Z</atom:updated>
            <content:encoded><![CDATA[<p>A new version of Zelig (5.1–2) is now available for download from CRAN. This release includes a lot of documentation and stability improvements. For example, setx and sim now fail informatively for ZeligEI models (these aren’t relevant for ecological inference model types) and from_zelig_model now works for models that use vglm.</p><h4>Odds Ratios</h4><p>There is an odds_ratios argument you can pass to summary with logistic regression models to show odds ratios rather than raw coefficient estimates. For example:</p><pre>library(zeligverse)<br>data(turnout)</pre><pre>z.out1 &lt;- <a href="http://docs.zeligproject.org/reference/zelig.html">zelig</a>(vote ~ age + race, model = &quot;logit&quot;, data = turnout)</pre><pre>summary(z.out1, odds_ratios = TRUE)</pre><p>This returns:</p><pre>Model:</pre><pre>Call:<br>stats::glm(formula = vote ~ age + race, family = binomial(&quot;logit&quot;), <br>    data = as.data.frame(.))</pre><pre>Deviance Residuals: <br>    Min       1Q   Median       3Q      Max  <br>-1.9268  -1.2962   0.7072   0.7766   1.0723</pre><pre>Coefficients:<br>            Estimate (OR) Std. Error (OR) z value Pr(&gt;|z|)    <br>(Intercept)         1.039           1.194   0.217 0.828325    <br>age                 1.011           1.003   3.689 0.000225 ***<br>racewhite           1.907           1.144   4.800 1.58e-06 ***<br>---<br>Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1</pre><pre>(Dispersion parameter for binomial family taken to be 1)</pre><pre>Null deviance: 2266.7  on 1999  degrees of freedom<br>Residual deviance: 2228.8  on 1997  degrees of freedom<br>AIC: 2234.8</pre><pre>Number of Fisher Scoring iterations: 4</pre><p>Thanks to <a href="https://github.com/adamobeng">Adam Obeng</a> for contributing much of the code for this feature.</p><h3>Breaking into Zelig</h3><p>The most substantial new feature in Zelig 5.1–2 is the ability to pass a fitted model object from lm, glm, or svyglm (see currently: <a href="http://docs.zeligproject.org/articles/to_zelig.html#supported-model-types">supported models</a>) to setx without having to estimate the model using zelig first.</p><blockquote><em>You can now break into Zelig.</em></blockquote><p>For example, we could estimate a model using lm:</p><pre>m1 &lt;- lm(speed ~ dist, data = cars)</pre><p>and then pass the output into the Zelig workflow to simulate and plot the quantities of interest:</p><pre>plot(sim(setx(m1)))</pre><p>This allows you to have the full functionality of non-Zelig estimation functions, including post-estimation capabilities, while also allowing you to easily find and plot quantities of interest from your models.</p><p>This capability, while still experimental, is the last major piece of an <a href="https://medium.com/zelig-dev/zelig-and-the-outside-world-d698fae10c74">ongoing project</a> to give Zelig considerably more flexibility to interact with non-Zelig functions — allowing users to break into and out of Zelig.</p><h4>Details</h4><p>Today’s release introduces an internal function — to_zelig— within setx that attempts to convert a non-zelig fitted model object to Zelig. If it is successful, it sets fitted values for the model as usual. These can then be used to simulate quantities of interest. This functionality builds on the from_zelig_model and zelig_qi_to_df functions introduced in previous recent releases. These functions enable users to break out of Zelig to take advantage of non-Zelig post-estimation functions and plotting packages such as <a href="http://ggplot2.tidyverse.org/reference/">ggplot2</a>. Here is a summary of the many different ways you can now interact with Zelig to effectively fit models and explore relevant quantities of interest:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*LABNmNeJMBVvoyG7S0mi2w.png" /><figcaption>Traditional zeligverse process (green) and interaction with outside of the zeligverse</figcaption></figure><h3>New Team Members</h3><p>We are also very happy to announce that over the summer the Zelig project will have two interns (Ben Sabath and John McCool). They’ve already helped improve the <a href="https://github.com/IQSS/Zelig/pull/274">readability</a> and <a href="https://github.com/IQSS/Zelig/pull/276">accuracy</a> of the Zelig documentation and have lots of big plans for the summer!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d17738cf6ef1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/breaking-into-zelig-5-1-2-released-and-new-team-members-d17738cf6ef1">Breaking into Zelig (5.1–2 released) and new team members</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zelig 5.1–1: New Website, zeligverse, Instrumental Variables Regression, Formula Updating, and More!]]></title>
            <link>https://medium.com/zelig-dev/zelig-5-1-1-new-website-zeligverse-instrumental-variables-regression-formula-updating-and-more-fc19c5ad1a8c?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/fc19c5ad1a8c</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[rstats]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Fri, 12 May 2017 13:16:09 GMT</pubDate>
            <atom:updated>2017-05-12T13:16:09.061Z</atom:updated>
            <content:encoded><![CDATA[<p>Zelig version 5.1–1 is now available for download from <a href="https://cran.r-project.org/package=Zelig">CRAN</a>!</p><h3>New website</h3><p>Perhaps the flashiest change is a new website for Zelig’s documentation: <a href="http://docs.zeligproject.org/">http://docs.zeligproject.org/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SPTSo5ZhZ0HevXjUYv5yLQ.png" /></figure><p>This site is your first port of call for detailed information about Zelig.</p><p>More important than being shiny and new, the website is constructed with <a href="https://medium.com/u/2f166aac6770">Hadley Wickham</a>’s <a href="https://github.com/hadley/pkgdown">pkgdown</a> tool for documenting R packages. This should make the documentation’s examples more robust and easier to maintain and easier.</p><p>Please send us any feedback you have on how to make the site more useful.</p><h3>zeligverse</h3><p>Most Zelig models are in the core <strong>Zelig</strong> package. However, a number of models are in other packages that you need to install separately from CRAN. This includes <a href="https://cran.r-project.org/package=ZeligChoice">ZeligChoice</a> for a range of categorical outcome models and <a href="https://cran.r-project.org/package=ZeligEI">ZeligEI</a> for ecological inference.</p><p>The main reason that Zelig is split into multiple packages is to ensure that it is robust to breaking changes in its models’ package dependencies. For example, if the <a href="https://cran.r-project.org/package=eiPack">eiPack</a> package underlying ecological inference models breaks, you can still load all non-EI models, because they are in separate packages.</p><p>However, as a user you don’t really need to think about this. Separately installing multiple packages and remembering which model is in which package is a pain.</p><p>So we’ve simplified the process with a new package (now on CRAN) called <a href="https://cran.r-project.org/package=zeligverse"><strong>zeligverse</strong></a>.</p><p>When you install zeligverse</p><pre>install.packages(&quot;zeligverse&quot;)</pre><p>the package installs all of the stable Zelig packages (if they aren’t already installed.</p><p>When you load zeligverse</p><pre>library(zeligverse)</pre><p>all of the Zelig packages will be loaded. You can also still load each Zelig package separately.</p><p>In the near future, we intend to expand the number of packages in zeligverse. The <a href="https://gking.harvard.edu/amelia">Amelia</a>, <a href="https://gking.harvard.edu/matchit">MatchIt</a>, and <a href="https://gking.harvard.edu/whatif">WhatIf</a> packages will be included. This will allow you to easily install and load all of the packages you need for a full Zelig workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*2CHk3T6sUfzfh02q3dSdTA.png" /><figcaption><a href="https://r.iq.harvard.edu/docs/tmp_bak/zelig.pdf">Imai, King, and Lau (2009, 30)</a></figcaption></figure><h3>Instrumental variable regression</h3><p>Due to <a href="https://github.com/IQSS/Zelig/issues/223">popular demand</a> we’ve added i<a href="https://en.wikipedia.org/wiki/Instrumental_variable#Interpretation_as_two-stage_least_squares">nstrumental variable (two-stage least squares) regression</a> to Zelig. You can access it in a <em>zelig</em> call by setting <em>model = ivreg</em>. It uses | syntax to separate the first and second stage equations. For example:</p><pre>z.out &lt;- <a href="http://docs.zeligproject.org/reference/zelig.html">zelig</a>(Y ~ X1 + X2 | Z1 + Z2 + Z3, model = &quot;ivreg&quot;, <br>               data = mydata)</pre><p>Where <em>X1</em> and <em>X2</em> are the regressors (in the second stage) and <em>Z1</em>, <em>Z2</em>, and <em>Z3</em> are the instruments (in the first stage).</p><p>One caveat: if you want to log (or otherwise transform) regressors, you need to do this <em>before</em> estimating the model. Imagine, we have a data set called <em>CigarettesSW </em>and it has two regressors that we want to natural log transform — <em>rprice </em>and <em>rincome. </em>First transform the variables:</p><pre>library(zeligverse)</pre><pre>data(&quot;CigarettesSW&quot;)</pre><pre>CigarettesSW$log_rprice &lt;- log(CigarettesSW$rprice)<br>CigarettesSW$log_rincome &lt;- log(CigarettesSW$rincome)</pre><p>Then estimate the model:</p><pre>z.out1 &lt;- <a href="http://docs.zeligproject.org/reference/zelig.html">zelig</a>(log(packs) ~ log_rprice + log_rincome |<br>                    log_rincome + tdiff + I(tax/cpi),<br>                    data = CigarettesSW,<br>                    model = &quot;ivreg&quot;)</pre><p>(We intend to support variable transformations in the <em>zelig </em>call for <em>ivreg </em>in a future version.)</p><h3>Formula updating and more</h3><p>You can now update your model estimation formulas using R’s <em>update</em> function and also its dot notation. <em>update</em> allows you to add variables to a model sequentially without having to retype the whole formula.</p><p>For example:</p><pre>library(zeligverse)<br>data(tobin)</pre><pre><br># define model implicitly </pre><pre>controls &lt;- ~ quant<br>z.out &lt;- zelig(formula = update(controls, durable ~ age + .), <br>                data = tobin, model = &quot;tobit&quot;)</pre><p>In this example, the <em>quant </em>variable is added to the formula at the position of the “.”.</p><p>For all model types (other than <em>ivreg)</em> you can now make arbitrary log transformations (e.g. <em>log2(x)</em>) and arbitrary <em>factor </em>calls within the <em>zelig</em> model call.</p><p>Please see the Zelig <a href="https://github.com/IQSS/Zelig/blob/master/NEWS.md#zelig-version-51-0">NEWS</a> for lots of other updates and improvements.</p><p>As always, we would love to hear your feedback on what we can improve in future versions. Give us a shout on <a href="https://github.com/IQSS/Zelig/issues">GitHub</a> (for reporting bugs and feature requests) and <a href="https://groups.google.com/forum/#!forum/zelig-statistical-software">Google Groups</a> (for general questions).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc19c5ad1a8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/zelig-5-1-1-new-website-zeligverse-instrumental-variables-regression-formula-updating-and-more-fc19c5ad1a8c">Zelig 5.1–1: New Website, zeligverse, Instrumental Variables Regression, Formula Updating, and More!</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zelig version 5.0–17: counterfactuals, imputations, nonlinear transformations]]></title>
            <link>https://medium.com/zelig-dev/zelig-version-5-0-17-counterfactuals-imputations-nonlinear-transformations-3b0c257fc63b?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/3b0c257fc63b</guid>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Tue, 21 Mar 2017 21:19:08 GMT</pubDate>
            <atom:updated>2017-03-21T21:18:01.138Z</atom:updated>
            <content:encoded><![CDATA[<p>We released Zelig version 5.0–17 to <a href="https://cran.r-project.org/package=Zelig">CRAN</a> the other day. There are lots of updates that make working with nonlinear covariate transformations, imputations, and counterfactuals much easier. Here are some highlights:</p><h3>Non-linear covariate transformations</h3><p>One of the most useful use cases for the Zelig method to simulating and displaying quantities of interest is when we have complex non-linear covariates that are expressed using multiple terms. Individual term coefficients for a polynomial, for example, are not interpretable individually. Quantities of interest simulated by Zelig, however, are interpretable in the normal way. For example, imagine we want to include a polynomial transformation of a variable in a linear model, e.g.:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*mPSf1GGFrIJNZEE9Z7_hZg.png" /></figure><p>To do this with Zelig v5.0–17, first create the squared variable values. Let’s use the standard R <em>cars</em> dataset and create a new variable of <em>dist</em> (distance) squared:</p><pre>cars.poly &lt;- cbind(cars, I(cars$dist^2))<br>names(cars.poly)[ncol(cars.poly)] &lt;- &#39;dist_2&#39;</pre><p>Now we estimate the model, simulate and plot the quantities of interest (which here is the expected <em>speed </em>given specific <em>dist </em>distances)</p><pre>library(Zelig)<br>library(dplyr)</pre><pre># Find range of dist to use as fitted values for simulating QI<br>dist_range &lt;- min(cars.poly$dist):max(cars.poly$dist)</pre><pre># Estimate, simulate, and plot<br><a href="https://christophergandrud.github.io/Zelig/reference/zelig.html">zelig</a>(speed ~ dist + dist_2, data = cars.poly, model = &#39;ls&#39;, <br>      cite = FALSE) %&gt;%<br>    <a href="https://christophergandrud.github.io/Zelig/reference/setx.html">setx</a>(dist = dist_range, dist_2 = dist_range^2) %&gt;%<br>    <a href="https://christophergandrud.github.io/Zelig/reference/sim.html">sim</a>() %&gt;%<br>    plot()</pre><p>Note that in <em>setx</em> we included both the untransformed <em>dist </em>fitted values<em> </em>and their square. The resulting plot is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9wSF0bfJKEW2950q2WRodA.png" /></figure><p>Use the same procedure for splines or other transformations.</p><h3>Counterfactuals</h3><p>Zelig allows you to explore quantities of interest for counterfactual scenarios. If we use fitted values from far outside the observed data that was used to estimate the model (extreme counterfactuals), these predictions will largely be speculation based on convenient, but hard to defend modeling assumptions. The following plot from <a href="http://gking.harvard.edu/files/abs/counterft-abs.shtml">Stoll, King,and Zeng (2006)</a> shows an example of two different model specifications — linear and quadratic — that both have good within-sample fit, but would lead to vastly different out-of-sample predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*lzZYxWDGkkZOy82DfxaxQA.png" /><figcaption><a href="http://gking.harvard.edu/files/abs/counterft-abs.shtml">Stoll, King,and Zeng (2006, 133)</a></figcaption></figure><p>How can we assess how “extreme” the counterfactuals are that we set in Zelig? This is especially difficult to determine when our models have many covariates.</p><p>Stoll, King and Zeng created the <a href="https://cran.r-project.org/package=WhatIf">WhatIf</a> R package to find quantitative answers to this question. It starts from the assertion that there will be more predictive model dependence the further away a counterfactual is from the observed data. One measure of distance that they propose is whether the counterfactual is within the “<a href="https://en.wikipedia.org/wiki/Convex_hull">convex hull</a>” of observed data, or not. The convex hull is the smallest <a href="https://en.wikipedia.org/wiki/Convex_set">convex set</a> including the observed data. Counterfactuals outside of this set will require extrapolation from the model to make predictions and will therefore be more model dependent.</p><p>The <em>whatif </em>function in WhatIf version ≥1.5–8 finds whether counterfactuals that we have set using Zelig’s <em>setx </em>method are in- or outside of the convex hull. For example:</p><pre><strong>library</strong>(Zelig)<br><strong>library</strong>(WhatIf)<br><strong>library</strong>(dplyr)</pre><pre># Estimate model and set fitted values<br>out &lt;- zelig(Petal.Width ~ Petal.Length + Sepal.Length, data = iris, <br>                        model = &quot;ls&quot;, cite = FALSE) %&gt;%<br>                setx(z.out, Petal.Length = 1:10)<br><br><em># Assess how extreme the counterfactuals are</em><br>summary(whatif(data = out))</pre><p>Here we set the fitted values (counterfactuals) of <em>Petal.Length</em> to range from 1 to 10. <em>Sepal.Length</em> is set at its mean in all of the counterfactuals. In the observed data, <em>Petal.Length</em> only ranges from 1 to 6.9. We may also not observe units with one of these <em>Petal.Length</em>s and mean <em>Sepal.Length.</em></p><p><em>whatif </em>returns the following assessment:</p><pre>## <br>## Summary of Counterfactual Inference Analysis<br>## <br>## Call:  whatif(data = out)<br>## <br>## Total Number of Counterfactuals:  10<br>## <br>## Number of Counterfactuals in Convex Hull:  4<br>## <br>## Average Percent &#39;Nearby&#39;:  0.142<br>## <br>## Counterfactual in Convex Hull, True or False, and Percentage of Observed<br>## Data Points &#39;Nearby&#39; Counterfactual:<br>##  Counterfactual In Hull Percent Nearby<br>##   1             FALSE   0.140000000   <br>##   2             TRUE    0.140000000   <br>##   3             TRUE    0.153333333   <br>##   4             TRUE    0.340000000   <br>##   5             TRUE    0.426666667   <br>##   6             FALSE   0.213333333   <br>##   7             FALSE   0.006666667   <br>##   8             FALSE   0.000000000   <br>##   9             FALSE   0.000000000   <br>##  10             FALSE   0.000000000</pre><p>We can see that counterfactual scenarios 7 through 10 are outside of the convex hull. We knew that this would be the case because as no observations have <em>Petal.Length</em>s of 7 through 10. We also see that scenario 1 and 6 are also outside of the convex hull, because no observations with these values also had <em>Sepal.Length</em>s close to the mean.</p><blockquote>It is a good idea to check if you are finding quantities interest for extreme counterfactuals whenever you use Zelig.</blockquote><h3>Summarising results with imputed data</h3><p>Zelig has been for some time well integrated with the <a href="http://gking.harvard.edu/amelia">Amelia II</a> package for data imputation. Zelig’s estimate/set/simulate/plot workflow has been an effective way to explore and present quantities of interest from these models. One area that had been lacking was a way to combine coefficients and standard errors from these models and present a summary of them.</p><p>Now Zelig includes a <em>combine_coef_se </em>function that uses Rubin’s Rules to combine parameter estimates from models estimated with multiply imputed data. For example, here is the function working inside of the usual summary method:</p><pre><strong>library</strong>(mice)<br><strong>library</strong>(Amelia)<br><strong>library</strong>(Zelig)<br>imp &lt;- amelia(nhanes)<br>mod &lt;- zelig(age ~ hyp, data = imp, model = &#39;poisson&#39;)<br>summary(mod)</pre><pre>## Model: Combined Imputations<br>##<br>##             Estimate Std.Error z value Pr(&gt;|z|)<br>## (Intercept)   -0.118     0.484   -0.24     0.81<br>## hyp            0.528     0.343    1.54     0.12<br>##<br>## For results from individual imputed datasets, use summary(x, subset = i:j)<br>## Next step: Use &#39;setx&#39; method</pre><p>Based on this new functionality, i<a href="https://github.com/leifeld/texreg/pull/29">mprovements</a> are being made to the <a href="https://cran.r-project.org/package=texreg">texreg</a> package to allow you to easily convert this type of output to LaTeX and HTML for dynamic documentation creation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b0c257fc63b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/zelig-version-5-0-17-counterfactuals-imputations-nonlinear-transformations-3b0c257fc63b">Zelig version 5.0–17: counterfactuals, imputations, nonlinear transformations</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zelig and the Outside World]]></title>
            <link>https://medium.com/zelig-dev/zelig-and-the-outside-world-d698fae10c74?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/d698fae10c74</guid>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ggplot2]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Mon, 27 Feb 2017 20:58:20 GMT</pubDate>
            <atom:updated>2017-02-27T20:58:19.691Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The functionality described in this post was inspired by conversations I had with <a href="https://github.com/izahn">@izahn</a> and <a href="https://github.com/vincentarelbundock">@vincentarelbundock</a> over on the <a href="https://github.com/IQSS/Zelig/issues/189">Zelig GitHub Issues page</a>. The complete feature set will be available from Zelig 5.0–16.</blockquote><p>Zelig is really good at estimating a wide variety of statistical models and returning plots of substantively meaningful quantities of interest. For many purposes, it is an ‘all-in-one’ solution for your statistical modeling needs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*CfLRHCRuXjXQhkKTPzzCUA.png" /><figcaption>Typical Zelig Workflow</figcaption></figure><h3>Breaking Out</h3><p>There are instances, however when you may want to extract contents of the Zelig Reference Class object. For example, you may want to run post-estimation functions on your regression results that Zelig doesn’t support. You also might want to take the simulations from <em>sim</em> and produce custom plots that, for example, are formatted in the same way as the other plots in your work.</p><p>To this end, Zelig versions <a href="https://github.com/IQSS/Zelig/blob/master/NEWS.md#zelig-version-50-15">5.0–15</a> and <a href="https://github.com/IQSS/Zelig/blob/master/NEWS.md#zelig-version-50-16">5.0–16</a> introduce three new functions:</p><ul><li><em>from_zelig_model</em>: extract a fitted model object from Zelig.</li><li><em>zelig_qi_to_df</em>: extract simulated quantities of interest from a Zelig object and return a <a href="http://vita.had.co.nz/papers/tidy-data.pdf">tidy formatted</a> data frame.</li><li><em>qi_slimmer</em>: a convenience function for returning only the bounds of the simulated quantity of interest distribution’s (95% by default) central interval and median. This is useful for efficiently plotting the distribution.</li></ul><p>These functions expand the possible Zelig workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*aGP1JEeLk9M_v_bwVzuURw.png" /><figcaption>Expanded Zelig Workflow</figcaption></figure><p>Let’s look at a simple example that incorporates <a href="https://cran.r-project.org/package=stargazer">stargazer</a> and <a href="http://docs.ggplot2.org/current/">ggplot2</a> into the Zelig workflow for creating output tables and custom plots [NOTE this example uses Zelig 4 wrappers to enable <a href="https://www.r-statistics.com/2014/08/simpler-r-coding-with-pipes-the-present-and-future-of-the-magrittr-package/">piping</a>].</p><p>Here is the traditional estimation/simulation/plotting workflow with piping (note the <em>data = . </em>in the <em>zelig </em>call tells the pipe to enter the <em>swiss</em> data object as the value of <em>zelig</em>’s data argument):</p><pre><strong>library</strong>(Zelig)<br><strong>library</strong>(tidyverse)<br><br>swiss %&gt;% <br>    <strong>zelig</strong>(Fertility ~ Agriculture + Examination, model = &#39;ls&#39;, <br>          data = ., cite = FALSE) %&gt;%<br>    <strong>setx</strong>(Agriculture = <strong>seq</strong>(1, 90, by = 5)) %&gt;%<br>    <strong>sim</strong>() %&gt;% <br>    <strong>plot</strong>()</pre><p>Producing this plot of the expected values of the <em>Fertility </em>variable across a range of fitted values of the <em>Agriculture </em>variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*yq3j15i8yvykUfx9Rn_agA.png" /></figure><p>We can use <em>from_zelig_model </em>to extract the Zelig fitted model object for creating a well formatted table with stargazer:</p><pre><strong>library</strong>(stargazer)</pre><pre># Estimate model<br>z_out &lt;- swiss %&gt;% <br>            zelig(Fertility ~ Agriculture + Examination, <br>                  model = &#39;ls&#39;, data = ., cite = FALSE)</pre><pre># Extract fitted model object and pass to stargazer<br>z_out %&gt;% from_zelig_model() %&gt;% stargazer(type = &#39;html&#39;)</pre><p>which produces:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*pC1VK2xeqtknlBBOLtTdKA.png" /></figure><p>We could then simulate quantities of interest and plot the results in ggplot2 by extracting the simulations from <em>sim</em>:</p><pre>z_sims &lt;- z_out %&gt;%<br>            setx(Agriculture = seq(1, 90, by = 5)) %&gt;%<br>            sim() %&gt;%<br>            zelig_qi_to_df()</pre><p>Before plotting the contents of <em>z_sims</em>, let’s look at its contents:</p><pre>head(z_sims)</pre><pre>setx_value Agriculture Examination expected_value predicted_value<br>         x           1    16.48936       75.41286        73.09981<br>         x           1    16.48936       70.84823        72.29154<br>         x           1    16.48936       81.95591        83.42378<br>         x           1    16.48936       73.78480        63.70319<br>         x           1    16.48936       72.03789        89.12260<br>         x           1    16.48936       78.88772        81.69652</pre><p><em>zelig_qi_to_df </em>returns a tidy-formatted data frame that includes every simulation, information about the scenario used in the simulation and the expected and predicted values that were simulated. In this example <em>setx_value </em>tells you if the simulation was from the first call to <em>setx</em> (denoted <em>x) </em>or <em>setx1 </em>(denoted <em>x1</em>), which can be useful for finding first differences. The <em>Agriculture </em>and <em>Examination</em> columns include the fitted values of the covariates in each scenario. The <em>expected_value </em>and <em>predicted_value </em>columns include the expected and predicted quantities of interest.</p><p>To plot the results using ggplot2 we may want to just keep the bounds of the simulation distribution’s central interval and median. The new <em>qi_slimmer</em> function can do that for us:</p><pre>z_sims &lt;- qi_slimmer(z_sims)<br><br>head(z_sims)</pre><pre>setx_value Agriculture Examination qi_ci_min qi_ci_median qi_ci_max<br>         x           1    16.48936  66.52290     74.90752  83.74183<br>         x           6    16.48936  66.66747     74.42137  82.34888<br>         x          11    16.48936  66.88929     73.93656  81.01327<br>         x          16    16.48936  67.02726     73.42013  79.72060<br>         x          21    16.48936  67.15937     72.95474  78.59869<br>         x          26    16.48936  67.47481     72.42850  77.47887<br>         x          31    16.48936  67.79143     71.90908  76.38637<br>         x          36    16.48936  67.83211     71.43531  75.23768<br>         x          41    16.48936  67.91079     70.98414  74.19120<br>         x          46    16.48936  67.77431     70.49351  73.38077<br>         x          51    16.48936  67.47261     70.04170  72.71222<br>         x          56    16.48936  67.04125     69.52855  72.51215<br>         x          61    16.48936  66.08855     69.08957  72.32978<br>         x          66    16.48936  65.08618     68.70346  72.23563<br>         x          71    16.48936  63.81562     68.25831  72.41644<br>         x          76    16.48936  62.56110     67.79559  72.67422<br>         x          81    16.48936  61.34663     67.36840  72.88702<br>         x          86    16.48936  60.15070     66.84647  73.15962</pre><p>The columns <em>qi_ci_min</em>, <em>qi_ci_median</em>, and <em>qi_ci_max</em> for the central interval of the simulations’ expected value from each scenario. You can choose expected value (<em>ev</em>, the default) or predicted value (<em>pv</em>) with <em>qi_slimmer</em>’s <em>qi_ci_type</em> argument.</p><p>By default <em>qi_ci_min</em> and <em>qi_ci_max</em> contain central interval bounds at the lower 2.5 and upper 97.5 percentiles, respectively. <em>qi_ci_median</em> contains the simulated distribution’s median. The interval can be changed with <em>qi_slimmer</em>’s ci argument.</p><p>It is then straightforward to plot this data frame using ggplot2:</p><pre>ggplot(z_sims, aes(Agriculture, qi_ci_median)) +<br>    geom_ribbon(aes(ymin = qi_ci_min, ymax = qi_ci_max), <br>                alpha = 0.3) +<br>    geom_line() + <br>    ylab(&#39;Expected Fertility&#39;) +<br>    theme_bw()</pre><p>producing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ittW_UgyTef7qOurMQ8XwQ.png" /></figure><h3>Breaking In</h3><p>From Zelig version <a href="https://github.com/IQSS/Zelig/blob/master/NEWS.md#zelig-version-50-14">5.0–14</a> there is a function called <em>to_zelig_mi</em> that allows you to take data frames created from a non-Amelia multiple imputation function and insert them into a Zelig Workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*XkAcXwyq6OZwATvKu-IrXA.png" /><figcaption>Expanded Zelig Workflow with non-Amelia Imputed Data</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d698fae10c74" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/zelig-and-the-outside-world-d698fae10c74">Zelig and the Outside World</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[More and More Tests!]]></title>
            <link>https://medium.com/zelig-dev/more-and-more-tests-5c0cfdb2e465?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/5c0cfdb2e465</guid>
            <category><![CDATA[softw]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Wed, 08 Feb 2017 11:39:57 GMT</pubDate>
            <atom:updated>2017-02-08T11:39:56.795Z</atom:updated>
            <content:encoded><![CDATA[<p>Starting from about <a href="https://medium.com/zelig-dev/testing-zelig-b41961cbfab8#.ke7x858su">14% code coverage in Zelig 5.0–13</a>, and due in no small part to lots of work by James Honaker, Zelig will have over 83% code coverage <a href="https://codecov.io/gh/IQSS/Zelig/commit/76fd5078dad14af5e6c9a6d3abd74c2f75889ace/graphs">in version 5.0–15</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*D-twSUXBHm5DSddc2yfBYA.png" /><figcaption>Graph of Zelig code coverage as of 8 Feb. 2017, Image from <a href="https://codecov.io/gh/IQSS/Zelig">codecov</a></figcaption></figure><p>To a certain extent this overstates the robustness of our testing suite as we surely don’t cover every potential user input. But the testing first approach, <a href="https://medium.com/zelig-dev/testing-zelig-b41961cbfab8#.ke7x858su">laid out in a previous post</a>, has already proven fruitful for catching a number of bugs and improving Zelig’s robust development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c0cfdb2e465" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/more-and-more-tests-5c0cfdb2e465">More and More Tests!</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Quick Post on Factors]]></title>
            <link>https://medium.com/zelig-dev/a-quick-post-on-factors-a9360ecb3afa?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/a9360ecb3afa</guid>
            <category><![CDATA[zelig]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Wed, 08 Feb 2017 11:24:03 GMT</pubDate>
            <atom:updated>2017-02-08T11:23:53.338Z</atom:updated>
            <content:encoded><![CDATA[<p>Zelig has had a hard time with unordered factor-level independent variables recently. From 5.0–13 <em>setx</em> just <a href="https://github.com/IQSS/Zelig/issues/156">didn’t work with factors</a> at all, e.g.:</p><pre>library(Zelig)<br><br>z.out &lt;- zelig(Petal.Width ~ Petal.Length + Species, data = iris, model = &quot;ls&quot;)<br>x.out &lt;- setx(z.out, Petal.Length = 1:10)<br>s.out &lt;- sim(z.out, x.out)<br><br>## Error in simparam$simparam %*% t(mm) : non-conformable arguments</pre><p>Beyond this, Zelig has for a long time not worked with factor variables that are created within a <em>zelig</em> call. For example, it was not possible to do this:</p><pre>z.out2 &lt;- zelig(unem ~ gdp + trade + capmob + as.factor(macro$country), model = &quot;ls&quot;, data = macro)<br><br>x.US &lt;- setx(z.out2, country = &quot;United States&quot;)</pre><p>As creating a factor inside of the Zelig call would produce parameter labels for each factor level like <em>asFactor(United States),</em> <em>setx</em> would then not be able to find <em>country </em>and so would return an error.</p><p>The old workaround was to convert all variables to factors outside of the <em>zelig</em> call, but this is clunky and non-intuitive for many users.</p><h3>The fixes</h3><p>These problems should now be fixed in Zelig Version 5.0–15.</p><p>The first problem had to do with an issue in the code of a new capability that allowed users to fit independent variables at custom central tendencies with a <em>fn</em> argument. The solution was to hard code mode as the central tendency for unordered factor-level variables. This is really the only valid central tendency for such variables anyways.</p><p>The second issue was solved by inserting a <a href="https://github.com/IQSS/Zelig/blob/93328a81acaeb7fbc237fff89316347a75cecdc5/R/utils.R#L303-L349">new function</a> at the beginning of the <em>zelig </em>call that identifies if a user has called <em>as.factor</em> inside the <em>zelig </em>call. It then converts the variable to a factor in the data and removes the <em>as.factor </em>call from the formula. This allows users to do things like <em>setx(z.out2, country = “United States”)</em>.</p><p>Please let us know if you have any other problems working with factors in Zelig (I’m sure I probably missed some possibilities)! Next up in this area is (likely) a similar solution to working with <a href="https://github.com/IQSS/Zelig/issues/199">polynomials and splines</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9360ecb3afa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/a-quick-post-on-factors-a9360ecb3afa">A Quick Post on Factors</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Documenting Zelig]]></title>
            <link>https://medium.com/zelig-dev/documenting-zelig-31e7d2f6c11b?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/31e7d2f6c11b</guid>
            <category><![CDATA[zelig]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Tue, 24 Jan 2017 17:01:39 GMT</pubDate>
            <atom:updated>2017-01-24T17:01:29.786Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>This post refers to Zelig version 5.0-13</em></blockquote><p>In a previous post I discussed the goal to <a href="https://medium.com/@chrisgandrud/testing-zelig-b41961cbfab8#.uzxatgjby">improve Zelig’s tests</a>. A related goal that I’m also focusing on is improve Zelig’s documentation. Clear documentation enables:</p><ul><li>users to easily <strong>use</strong> Zelig in their work.</li><li>maintainers to more quickly understand and so <strong>maintain and improve the source code</strong>, e.g. the <a href="http://catb.org/esr/writings/taoup/html/ch01s06.html#id2878339">Rule of Robustness</a>: “robustness is the child of transparency and simplicity.”</li><li><strong>better unit and integration tests</strong>. Documentation specifies expected software behaviour that can guide the development of unit and integration tests. Also, documentation built using the <a href="https://en.wikipedia.org/wiki/Literate_programming">literate programming paradigm</a> can itself test software behaviour when compiled.</li></ul><h3><a href="https://github.com/IQSS/Zelig/issues/163">Dueling Zelig 4 and 5 documentation</a></h3><p>Zelig is currently documented in a number of ways. The general Zelig approach is documented in journal articles, including those <a href="http://gking.harvard.edu/files/making.pdf">HERE</a> and <a href="http://gking.harvard.edu/files/abs/z-abs.shtml">HERE</a>. Zelig syntax is documented using the package’s internal documentation (also <a href="https://cran.r-project.org/web/packages/Zelig/Zelig.pdf">available on CRAN</a>) and a <a href="http://docs.zeligproject.org/en/latest/">website</a>.</p><p>Currently, these sources provide “dueling documentation” between Zelig versions 4 and 5 syntax that may confuse some users.</p><p>Zelig 5 (released in 2015) introduced a substantial change to Zelig’s syntax and behaviour. The package now relies on R’s new (as of R version 2.12) <a href="http://adv-r.had.co.nz/R5.html">Reference Classes</a> (RC). RC have a number of key advantages for code development and readability. Reference classes use conventions that would be familiar to users of other object oriented programming languages such as Python.</p><p>The following code gives a sense of how Zelig uses reference classes to run a simple regression on R’s included <em>swiss</em> data set:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a06b5c5b1bf0d201319a3bb9edac7e53/href">https://medium.com/media/a06b5c5b1bf0d201319a3bb9edac7e53/href</a></iframe><p>Though the general steps are the same — <em>estimate, set fitted values, simulate, plot</em> — , the Zelig 5 RC syntax is different than previous versions. You probably noticed that RC objects are not overwritten by the assignment operator <em>&lt;- </em>but are instead mutated by it.</p><p>My suspicion is that most R users are unfamiliar with Reference Classes as, in my experience, not many packages use them, at least not at the user interface level. Therefore documentation, with lots of examples and explanation with Zelig’s Reference Class behaviour, is crucial for most people to be able to use the package.</p><p>Zelig 5 also has wrappers for its key functions that allow users to continue following Zelig 4 syntax. For example, a similar operation to the one we just saw using Zelig’s wrappers that imitate version 4’s syntax would be written:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b065dd43ebf45ce98f8244e5a3da92d4/href">https://medium.com/media/b065dd43ebf45ce98f8244e5a3da92d4/href</a></iframe><p>Using Reference Classes enables code development and readability improvements and having Zelig 4 wrappers enables compatibility with code written in previous versions. However, this state of affairs creates documentation challenges.</p><p>When users attempt to access Zelig’s help files on functions such as <em>zelig </em>or <em>sim</em> using R’s question function (<em>?</em>) they are typically directed to the documentation for the Zelig 4 wrapper. To access the Zelig 5 documentation they must invoke the <em>help</em> method on their RC object, e.g. <em>zls$help()</em>. This will launch the user’s web browser and take them to the relevant documentation on the <a href="http://docs.zeligproject.org/en/latest/index.html">Zelig website</a>. This difference could be confusing.</p><p>While the Zelig 4 wrappers and Zelig 5 methods have broadly similar functionality, they do occasionally differ in non-obvious ways. For example, notice in Zelig 4 wrapper code above the <em>x </em>and <em>x1 </em>fitted<em> </em>values are set within the same <em>setx</em> call, whereas there are separate Zelig 5 methods for these. The functionality of these two approaches also differs as Zelig 4 <em>setx</em> will accept a range of values for a variable, <a href="https://github.com/IQSS/Zelig/issues/164">while you need the <em>setrange </em>method in Zelig 5 syntax</a>.</p><p>Ideally, these different behaviours would be more clearly documented and accessible to users with little knowledge of Reference Classes.</p><h3><a href="https://github.com/IQSS/Zelig/issues/168">Updated Documentation Toolkit</a></h3><p>Zelig currently uses literate programming in the form of the <a href="http://www.sphinx-doc.org/en/1.5.1/">Sphinx</a> Python package and <a href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> to create its online documentation. There are two ways that this could be improved.</p><p>The first is to convert to using <a href="http://rmarkdown.rstudio.com/">R Markdown</a>, <a href="https://cran.r-project.org/web/packages/roxygen2/index.html">roxygen2</a>, and <a href="http://hadley.github.io/pkgdown/">pkgdown</a>. Zelig already uses roxygen2 to generate its internal documentation, though this contains more information about Zelig 4 and differs from the Zelig 5-focused online documentation. At the time the Zelig documentation was being developed, R Markdown was in its infancy and it was unclear how successful it would be. In the time since then, these tools have been widely embraced by the R developer community and they are fairly full featured.</p><p>pkgdown and R Markdown will enable:</p><ul><li>Close integration with the internal documentation, which is currently different from the online documentation in ways that are potentially confusing.</li><li>Closer test integration as the online docs could be tested every time Zelig goes through a build check. This would enable more accurate documentation (e.g. docs that don’t include errors) and provide another “test surface” for finding errors.</li><li>Simplify the package development toolchain.</li><li>Easier community contributions as Markdown is now widely used in the R community.</li></ul><h3>Documentation Build Error Handling</h3><p>An important part of this toolchain changes is to enable more robust documentation. Currently, when documentation is built by Sphinx. If there were errors the build would not break. Instead error messages are outputted into the documentation. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*sgvQX9xTZBfrwPUUHCEebA.png" /><figcaption>Zelig Documentation for probitsurvey as of 2017–01–16</figcaption></figure><p>Needless to say, this does not produce useful documentation. It also did not alert package maintainers of possible broken code that needed to be fixed.</p><p>While converting the documentation to R Markdown, I’m adding the <a href="https://yihui.name/knitr/options/">knitr global option</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5fb232608c02fe58c69e2fe630464db/href">https://medium.com/media/c5fb232608c02fe58c69e2fe630464db/href</a></iframe><p>This causes the build to fail if any of the code chunks return an error.</p><h3><a href="https://github.com/IQSS/Zelig/issues/165">Dynamically Built README file</a></h3><p>Some potential Zelig users will likely encounter the package via the <a href="https://github.com/IQSS/Zelig">GitHub site</a>. This could be a good place to inform users of crucial information, especially the new Zelig 5 syntax, and provide a quick start example. To ensure that the example runs without problems, we could also add <em>knitr::opts_knit$set(stop_on_error = 2L)</em>.</p><h3>Faster Crashes, More Informative Error Messages</h3><p>Developing a <a href="https://medium.com/@chrisgandrud/testing-zelig-b41961cbfab8#.uzxatgjby">fuller test suite</a> will help us identify more errors that users might trigger. When users are given an error, the error messages should be informative about what caused the problem and how to resolve it.</p><p>Error messages in Zelig good be improved on these points. For example, if you want to use Zelig 4 syntax, but forget to specify the <em>model</em> type then in version 5.0–13 you would receive the error:</p><blockquote>Error in models4[[model]] : invalid subscript type ‘symbol’.</blockquote><p>A more informative message would be something like:</p><blockquote>Estimation model type not specified.</blockquote><blockquote>Select estimation model type with the model argument.</blockquote><p>A related issue with the Reference Class Zelig implementation is that no error message is produced when a user skips or does not successfully complete a step in the <em>estimate, set, simulate, plot</em> Zelig workflow. For example, the following code is what we saw above, but omits the <em>setx</em> step that sets fitted values needed to find meaningful quantities of interest from the estimation model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef30735eec84e4f0ee50fd27fa0e71fb/href">https://medium.com/media/ef30735eec84e4f0ee50fd27fa0e71fb/href</a></iframe><p>No messages or errors are returned until the final line is called. The returned warning is:</p><blockquote>Warning message:<br>In par(old.par) : calling par(new=TRUE) with no plot</blockquote><p>There are two issues here:</p><ul><li>Zelig did not fail early enough.</li><li>Zelig did not fail informatively.</li></ul><p>A user might believe their problem is in the <em>graph</em> method and spend time trying to decipher the oblique error message. When in fact their <em>sim</em> call should have returned an informative error, preventing them from progressing to the <em>graph</em> call. If a user does not supply required information then the Zelig workflow should <a href="https://github.com/IQSS/Zelig/issues/167">fail immediately</a> and provide an informative warning or error message.</p><h3><a href="https://github.com/IQSS/Zelig/issues/162">NEWS</a></h3><p>Many R packages include a plain text file called <em>NEWS</em> that documents in detail all of the changes made in a given release. These files are include on the package’s CRAN page. Currently Zelig lacks such a file. It would be useful to add a NEWS file to update users about changes, especially breaking changes, made in Zelig releases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31e7d2f6c11b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/documenting-zelig-31e7d2f6c11b">Documenting Zelig</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing Zelig]]></title>
            <link>https://medium.com/zelig-dev/testing-zelig-b41961cbfab8?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/b41961cbfab8</guid>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[zelig]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Tue, 24 Jan 2017 16:45:44 GMT</pubDate>
            <atom:updated>2017-01-27T18:22:33.487Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>This post refers to Zelig version 5.0–13</em></blockquote><p>One of Zelig’s main selling points is that it provides a common syntax for estimating models from a wide range of R packages. This is also one of its key software maintenance challenges.</p><p>To provide a common estimation syntax Zelig wraps many other user-created estimation packages such as <a href="https://cran.r-project.org/web/packages/survey/index.html">survey</a>, <a href="https://cran.r-project.org/web/packages/MCMCpack/index.html">MCMCpack</a>, <a href="https://cran.r-project.org/web/packages/VGAM/index.html">VGAM</a>, among others. Depending on these packages opens Zelig up to breaking whenever they are updated. For example, a small change in accepted argument values for an estimation function from a given package could break Zelig users’ code for those models.</p><p>In the past, with an exception discussed below, there hasn’t been a systematic way for maintainers to known about such breaks, unless a user reports them. Effectively reporting code breaks has often been difficult for users because the error messages that are returned to them are often oblique without an intimate understanding of Zelig’s and the wrapped package’s internals.</p><p>Perhaps the best solution to these problems is to build an <em>extensive </em><a href="https://en.wikipedia.org/wiki/Unit_testing"><em>unit and integration test</em></a><em> suite</em> that is run at <em>frequent intervals</em> using the most recent CRAN releases of Zelig’s dependencies. These will allow broken code to be identified and fixed early by Zelig’s maintainers (for a general overview see <a href="http://r-pkgs.had.co.nz/tests.html">Hadley Testing R Packages</a>). More informative error messages will also help users understand Zelig’s limitations as well as debug and report errors.</p><p>Here are the tools and strategies I’m currently working with to achieve these goals. As always, feedback and suggestions are highly appreciated.</p><h3>Starting Point</h3><p>Zelig already contains a number of <a href="http://homes.cs.washington.edu/~borning/papers/sevcikova-issta-2006.pdf">unit tests for Zelig’s stochastic models</a>. These have largely <a href="https://github.com/IQSS/Zelig/commit/f9a592ed336bd944b38b62ef3029aea287b0d186">been in place since late 2015</a>. Zelig allows us to present and explore substantively meaningful quantities of interest via parameter estimate simulations. This presents a testing challenge. Unit tests compare a predefined expectation — a value, an error message, and so on — to the output of a test call. Many of Zelig’s outputs are stochastic, not deterministic. So it’s difficult to predetermine what result we expect from them even given the same inputs. Monte Carlo unit tests overcome this problem by using an algorithm that will be published in future work. These tests are implemented with the <em>mcunit.test</em> method. Currently there are Monte Carlo unit tests for logistic, log-normal, least squares, negative binomial, poisson, and probit models.</p><h3>Increasing Test Coverage</h3><p>As of version 5.0–13 these tests cover about 14% of the Zelig package source code. This calculation was made using the tools discussed below.</p><p>Ideally, tests will cover closer to 100% of Zelig’s code. A caveat before proceeding: this is a substantial, but also potentially somewhat superficial goal to achieve. Just because a unit of code is called by a test does not mean that that test compares the output to a substantively meaningful expectation that would give an indication of problems that real-world Zelig users may encounter. Also traditional code coverage statistics do not include coverage of Zelig dependencies that could be invoked.</p><p>So 100% code coverage is just one measure of how well Zelig is tested. In addition, tests should be written to capture the full range of user behaviour (both intended and unintended).</p><h3>Unit and Integration Tests</h3><p><a href="https://en.wikipedia.org/wiki/Unit_testing">Unit tests</a>, as the name implies, should be designed to test a discrete code unit. This makes results from the tests easier to interpret and debug if expectations are not met as we can begin looking for problems in specific code units. Unit tests are also less redundant and so more computationally efficient than broad overlapping tests.</p><p>The Zelig workflow involves a series of methods — <em>new</em>, <em>zelig</em>, setx, <em>sim</em>, <em>graph </em>— that need to work together<em>.</em> So, in addition to unit testing, we should test the whole workflow — <a href="https://en.wikipedia.org/wiki/Integration_testing">integration testing</a>. These add to the tests’ computation time as they will test functionality conducted by unit tests. However they are crucial for a package like Zelig where each piece needs to work together in order for users to successfully use it into their research.</p><h3>Testing Tools</h3><p>I’m currently implementing a set of testing and continuous integration tools that have become fairly widely used in the R community and beyond. These include:</p><ul><li>The <a href="https://cran.r-project.org/web/packages/devtools/index.html">devtool/</a><a href="https://cran.r-project.org/web/packages/testthat/index.html">testthat</a> R packages for package building and development-time testing.</li><li>Custom assertive functions (stored in <em>R/assertions.R</em>) for runtime testing.</li><li><a href="https://travis-ci.org/IQSS/Zelig">Travis CI</a> for continuous integration testing on Linux/macOS.</li><li><a href="https://ci.appveyor.com/project/christophergandrud/zelig">AppVeyor</a> for continuous integration testing on Windows.</li><li>covr/<a href="https://codecov.io/gh/IQSS/Zelig">codecov</a> for reporting and exploring the coverage of development-time tests.</li></ul><p>Here is a summary of the development-time test coverage in Zelig 5.0–13:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*r4P2Q5Q5KPQMI_hPc8OyuQ.png" /><figcaption>Zelig code coverage in Zelig 5.0–13(from <a href="https://codecov.io/gh/IQSS/Zelig/commit/ffc59d0fcd91ea7e659129cf2001631994bb0557/graphs/icicle.svg">Codecov</a>). Green areas represent code covered by a unit test.</figcaption></figure><p>This plot shows the amount of code that is tested as of the most recent GitHub master Zelig build at the end of 2016. Just under 15% of the code is invoked by one of the stochastic unit tests. Ideally, Zelig will have 70–100% code coverage.</p><h3>Informative Error Messages &amp; Runtime Testing</h3><p>In a separate post on documentation, I’m detailing plans for building out Zelig’s documentation. One component of this that is intimately tied with unit tests is creating informative warning and error messages.</p><p>Frequently, the focus of unit testing is <strong>making sure that software requirements work [REQUIRE TEST]</strong>. Zelig’s Monte Carlo tests are requirement tests. A clear requirement of a Zelig model class is that it makes parameter estimates and simulated quantities of interest in an expected way. I.e. given a known data generating process, are the statistical model’s outputs within the expected distribution?</p><p>Unit tests should also test the software’s limitations. If software is going to fail, because for example a user has omitted a required input, it should fail early and informatively so that the user can easily learn what went wrong and how to fix it. This requires a different type of unit test: <strong>testing for failure [FAIL TEST]</strong>. Failure testing follows two principals:</p><ul><li>If the software is going to fail, fail as soon as possible.</li><li>When it fails, return a useful message that is understandable to moderately skilled and even novice users.</li></ul><p>A major cause Zelig failure is that users pass data to Zelig functions that Zelig and the packages it wraps do not support. An important component of this is the use of <strong>runtime</strong> <strong>assertive</strong> functions. Runtime testing involves checking the state of data to make sure that it conforms to supported formats. It is easy enough in R to build such tests using base R code. For example, we can test if an object called <em>x</em> is scalar and return an informative error message if not with: <em>if(length(x) != 1) stop(‘x must be scalar’)</em>. The assert that package just makes this code somewhat more compact and returns consistent informative error messages, i.e. package maintainers have to spend less time remembering if there is a similar, but slightly different error message for the same problem elsewhere in the package.</p><p>There are a number of assertive runtime testing packages — e.g. <a href="https://github.com/hadley/assertthat">assertthat</a>, <a href="https://cran.r-project.org/web/packages/assertive/index.html">assertive</a>. Given Zelig’s use of reference classes, which are not well supported by these packages, a desire to minimise dependencies, and a need to have highly context specific messages most of Zelig’s assertions will be built in house. They will follow assertive’s <em>is_*</em> syntax.</p><h3>Development, Testing, and Release Workflow</h3><p>The (modified) <a href="https://en.wikipedia.org/wiki/Continuous_integration">continuous integration</a> workflow that I’m implementing these tools within goes like this (note: subject to change):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*bft80noVjPOKIDlIdELfNg.png" /><figcaption>Proposed Zelig Continuous Integration Workflow</figcaption></figure><p>Everything starts from the <a href="https://github.com/IQSS/Zelig">master branch</a> stored on GitHub. The first step to creating a new feature or bug fix is to checkout a new branch for that feature/hotfix from master. Before diving into building the feature/fix, the next step is to create require and fail tests using testthat for the expected behaviour. After this, the feature/fix is coded and tested locally using the <em>test </em>function from the devtools package. This builds the package and runs all tests including executable documentation examples. If the tests pass, then the feature/hotfix branch is merged back into master. If not, then more work is needed to create code that passes the test.</p><p>The master branch is built and tested by the cloud continuous integration services Travis CI and Appveyor. This occurs whenever commits are made to master and, in the case of Travis CI every week. The weekly build tests if Zelig is still compatible with the newest releases of its dependencies. Currently the weekly build is only conducted on Travis CI because this is a free (beta) feature for public GitHub repositories like Zelig.</p><p>While I work on increasing Zelig’s code coverage, I’m also following this workflow, simply considering each test a new feature, thus collapsing the Create Test and Create Feature steps into one.</p><h3>Test File Organisation</h3><p>Presently I’m in the process of creating tests primarily organised around each R source code file in the <a href="https://github.com/IQSS/Zelig/tree/master/R">R sub-directory</a>. E.g. for each source file in the directory, I create a new git branch and then a new <em>tests/testthat/test-*.R</em> file with tests of that source file. Once the tests are built and pass, the branch is merged into master. While I’m going through each of these files one-by-one, I’m also using <a href="https://yihui.name/formatR/">formatR</a> to ensure that the Zelig source code follows a consistent format and so will hopefully be (marginally) easier to debug.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b41961cbfab8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/testing-zelig-b41961cbfab8">Testing Zelig</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rules of Thumb for Robust Zelig Development: an evolving/non-exhaustive list]]></title>
            <link>https://medium.com/zelig-dev/rules-of-thumb-for-robust-zelig-development-an-evolving-non-exhaustive-list-5a420ffdc8fa?source=rss----c6cfd6af8596---4</link>
            <guid isPermaLink="false">https://medium.com/p/5a420ffdc8fa</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Christopher Gandrud]]></dc:creator>
            <pubDate>Thu, 19 Jan 2017 20:33:54 GMT</pubDate>
            <atom:updated>2017-01-19T20:38:16.173Z</atom:updated>
            <content:encoded><![CDATA[<p>The title explains it all. Many of these are taken from with slight modifications from <a href="http://catb.org/esr/writings/taoup/html/ch01s06.html#id2878339">The Unix Philosophy.</a></p><p>Get in touch if you have any suggestions.</p><ul><li><strong>Fail early and informatively!</strong> <strong>|</strong> Makes it easier to identify and solve problems.</li><li><strong>All code additions are accompanied by substantively meaningful unit tests that pass</strong>, i.e. tests of both stochastic output (when relevant) and the full range of possible user inputs. <strong>|</strong> Will result in fewer bugs, better code structure, and so more reliable software.</li><li><strong>Think like a novice user | </strong>One of Zelig’s key selling points is that it makes hard things easy. So Zelig behaviour, documentation, and messages including error messages should be intelligible to novice users as much as possible.</li><li><strong>Document</strong> <strong>everything | </strong>Clear and concise documentation is crucial for both users being able to actually use the software and maintainers to be able to maintain it over time.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a420ffdc8fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/zelig-dev/rules-of-thumb-for-robust-zelig-development-an-evolving-non-exhaustive-list-5a420ffdc8fa">Rules of Thumb for Robust Zelig Development: an evolving/non-exhaustive list</a> was originally published in <a href="https://medium.com/zelig-dev">Zelig-dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>