<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kilian Hekhuis on Medium]]></title>
        <description><![CDATA[Stories by Kilian Hekhuis on Medium]]></description>
        <link>https://medium.com/@kilianhekhuis?source=rss-f3f3765190c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*da7z21YiD55O99y3Uryk8w.jpeg</url>
            <title>Stories by Kilian Hekhuis on Medium</title>
            <link>https://medium.com/@kilianhekhuis?source=rss-f3f3765190c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:13:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kilianhekhuis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exposing a URL for downloading a file with OutSystems Reactive]]></title>
            <link>https://itnext.io/exposing-a-url-for-downloading-a-file-with-outsystems-reactive-f19f2f7345a9?source=rss-f3f3765190c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f19f2f7345a9</guid>
            <dc:creator><![CDATA[Kilian Hekhuis]]></dc:creator>
            <pubDate>Wed, 31 May 2023 10:45:11 GMT</pubDate>
            <atom:updated>2023-11-03T08:28:56.984Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This is an article about developing with the </em><a href="https://www.outsystems.com/"><em>OutSystems Platform</em></a><em>. If you are not yet developing with OutSystems, you should :).</em></p><p><em>Special thanks to </em><a href="https://medium.com/@fabiofantato"><em>Fábio Fantato</em></a><em>, </em><a href="https://medium.com/@remcodekkinga"><em>Remco Dekkinga</em></a><em>, Daniël Kuhlmann, Greg Witten, </em><a href="https://medium.com/@leonardo-monteiro-fernandes"><em>Leonardo Fernandes</em></a><em> and </em><a href="https://medium.com/@evertvanderzalm"><em>Evert van der Zalm</em></a><em> for proofreading and suggestions.</em></p><h3>Introduction</h3><p>You may have had to implement this use case using OutSystems: expose a URL that when accessed downloads a file. Many years ago, when the OutSystems Platform offered only what is nowadays called “Traditional Web”, this was pretty straightfoward: you created a Screen with a Preparation, and inside the Preparation you retrieved the file to download and ended the Preparation in a Download node.</p><p>However, for obvious reasons, this is no longer possible with Reactive: it doesn’t have Preparations, and runs client-side. So how to implement the download URL in Reactive? Since this question comes up <a href="https://www.outsystems.com/forums/">on the forum</a> from time to time, I’d thought I’d write a short article about what I consider a best practice to handle this.</p><h3>REST endpoint</h3><p>REST is a way to transfer data using the HTTP protocol (and typically wrapped as HTTPS for security). It is typically used for APIs, but since its underlying protocol is HTTP, we can also use it to serve other types of data. This makes it ideal for our purpose, returning a file as binary data. To do so, we first need to expose a REST API. If you’re not sure how to do this, be sure to check the <a href="https://success.outsystems.com/documentation/11/extensibility_and_integration/rest/expose_rest_apis/expose_a_rest_api/">official documentation</a>, but we’ll go through the steps below.</p><p>1. On the Logic tab of your module, right-click the REST item in the Integrations folder. The context menu shows as the first option “Expose REST API”. Select it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*Q8o5RBmAmrcqLcQaREhCcA.png" /><figcaption>Be sure to select the first option, as we want to Expose, not Consume.</figcaption></figure><p>2. Service Studio now asks for the name of your API. What you type here will be part of the URL, so choose the name wisely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*VW5NsXUgpmi5xdM24FrDBQ.png" /><figcaption>Do not call it “RESTAPI1”. Choose a better name, like “TheEmpireStrikesBack”.</figcaption></figure><p>3. Next, right-click the API you just created. The context menu shows “Add REST API Method” as the first option. Select it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*SMImGnyhnPir_0sgxsbpCA.png" /><figcaption>Don’t worry about the red squigglies, they will disappear once you add a method!</figcaption></figure><p>4. Service Studio now asks for the name of your method. Choose a descriptive name, like FileGet. Though this name will be, by default, part of the URL, we’ll change it later on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*ymrawSWI9wj-0lXeZ48e8g.png" /></figure><p>5. Assuming you want to be able to handle more than one file, you need to add a parameter that is a unique identifier for each specific file. This can be a numeric Id, or a Text identifier, or anything that will allow your code to retrieve the right file from the database (or elsewhere) for download. Since this is implementation-specific, I’ll assume a numeric Id here.</p><p>Also, you may want to have some sort of security or access control and add another parameter, like an API key, that you can verify. For simplicity, I’ll omit that here, but see the next section down below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*yKQyWkqFQA9MDe8KtscUJA.png" /></figure><p>6. To output the binary content of the file, we need an Output Parameter of type Binary Data. It should be in the Body of the response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*RT0-K9b4_oFyJBOIjesD6A.png" /></figure><p>7. Now we need to write some code. I’ll assume the file is in the database. For simplicity’s sake, I’ve created a simple Entity looking like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*gIK21tqfWXZ3U-0wIEXbog.png" /></figure><p>It has an Id, a filename, a content type (the MIME type to return) and the actual binary content. Of course, you can use any data model you like. Also, content type is optional, the default used by OutSystems is application/octet-stream (i.e. generic binary data).</p><p>8. In the REST method, add an Aggregate to retrieve the data from the database. In my implementation, the FileID is matched with the Id of the entity, but if you have another type of identifier, you’ll have to use that. Since we only want a single record, don’t forget to set the Max. Records property of the Aggregate to 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*_gQvefL-MDrFa-KblpJdXw.png" /></figure><p>9. Next, assign the binary content that’s returned from the Aggregate to the output variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*EomVIqxEEzHbIt_xWRzlPA.png" /></figure><p>10. To signal to a browser or other application that the binary content is a file, we need to add a header to the returned message, with the name “Content-Disposition”. The value of this header has the following format:</p><p><strong>attachment; filename=<em>filename</em></strong></p><p>where <em>filename</em> is the name of the file, as retrieved from the database. Since the OutSystems Platform does not add the Content-Disposition header itself, I’d normally add it as an Output Parameter with its “Send In” property to “Header”. However, since this doesn’t work for Content-Type (see next step), I’ll use the AddHeader action from the HTTPRequestHandler extension instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fpj1usOdWI4QA2MFltYg8Q.png" /></figure><p>11. To add the proper MIME type, we need to add another header, with the name “Content-Type”. Since Content-Type <em>is</em> a header that the Platform adds, and since it doesn’t recognize you already added a Content-Type header if you use the Output Parameter trick, to prevent two Content-Type headers in the returned message, we need to use the AddHeader action.</p><p>Content-Type has, as its value, only the MIME type, so we can directly assign that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*3EqilC-urt3AY-kgcnxdjg.png" /></figure><p>Our REST method should now look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/163/1*6l0pVcI1hdjOmDqNJllW1A.png" /><figcaption>Dark mode is evil. Do not join the Dark Side!</figcaption></figure><p>And that’s it! You now have created an endpoint that allows you to download a file, with the correct MIME type. To see the URL that the REST method exposes, select “Open Documentation” from the REST API’s context menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Kg4KhYhxbuVvPqdirn6Gbw.png" /></figure><p>The standard REST API Swagger will appear, showing the details of what you just created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jojE696-ahsVx8hJ4absPw.png" /></figure><h3>Security</h3><p>Now let’s address the elephant in the room: what about security? If you followed the steps in the previous section, you’ll have a nice URL that allows downloading files by ID, but it has two downsides:</p><ol><li>The ID is numerical, and someone might try to access the URL with incremental ID numbers to fetch everything that’s available, even when that’s not quite what you had in mind and;</li><li>Everyone can access the URL and download files, there’s no restriction on who can download what.</li></ol><p>These two points might not be a problem if you run your software on-premise, only internal company software can access the URL and the files that can be downloaded are restricted to non-confidential information (for example, a URL used to retrieve the company logo in various DPIs). But if you are running in the cloud, and you’re exposing sensitive information, you might want to mitigate potential problems. So let’s look at possible solutions.</p><h4>Using non-numerical IDs</h4><p>To mitigate the first security flaw, instead of using integers to identify a file, use something that can’t be easily guessed. This could be a GUID, a hash of a filename (or, if the filename needn’t be unique, the filename together with the file’s database Id) or something else that’s unique and opaque. You need to store this ID in the database, so you can search for it when it’s specified in the call to your REST method. If you have a large number of files that can be retrieved, be sure to add a unique database Index to the attribute containing the ID, so searching is fast.</p><h4>Restrict access to files</h4><p>There are many ways to secure a REST API. If you have a fully fledged API, you might want to use an <a href="https://medium.com/itnext/exposing-an-outsystems-rest-service-with-an-oauth-style-authorization-fded258cbe14">OAuth-style authorization</a> mechanism, but for our simple file download this might be a bit overkill. Generally speaking, you can handle this in two ways:</p><ol><li>Use the OutSystems built-in authentication mechanism, or;</li><li>Create your own authentication.</li></ol><p>The first method is described in the official OutSystems documentation in detail. Check <a href="https://success.outsystems.com/documentation/11/extensibility_and_integration/rest/expose_rest_apis/add_basic_authentication_to_an_exposed_rest_api/">this page </a>for Basic Authentication and <a href="https://success.outsystems.com/documentation/11/extensibility_and_integration/rest/expose_rest_apis/add_custom_authentication_to_an_exposed_rest_api/">this page</a> for Custom Authentication. Both make use of the OnAuthentication callback action. This is a good way to deal with authentication that makes use of header parameters, especially of you have many methods all having the same authentication mechanism and using the same headers, but perhaps a bit overkill for our scenario here. It is also not ideally suited for a scenario where <em>authorization</em> is also important, i.e. restricting or allowing access to certain resources (in our case: files) based on who is being authenticated, as the OnAuthentication action does not have access to non-header parameters.</p><p>The second method, and one I would recommend in this case, is adding an input parameter to the method, which contains some kind of API key or user identification. Like with the first mechanism, you can use it to authenticate the user (it must be a valid API key), but if you have different levels of authorization, you can also directly check whether this API key is allowed to access the specified resource (how to do this is out of scope for this article).</p><p>That wraps it up, I hope you found it an interesting read.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f19f2f7345a9" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/exposing-a-url-for-downloading-a-file-with-outsystems-reactive-f19f2f7345a9">Exposing a URL for downloading a file with OutSystems Reactive</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A SOAPy business — extending SOAP with OnBeforeRequestAdvanced (and then some)]]></title>
            <link>https://itnext.io/a-soapy-business-extending-soap-with-onbeforerequestadvanced-and-then-some-b6512dcfbc8?source=rss-f3f3765190c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6512dcfbc8</guid>
            <category><![CDATA[outsystems]]></category>
            <category><![CDATA[soap]]></category>
            <dc:creator><![CDATA[Kilian Hekhuis]]></dc:creator>
            <pubDate>Thu, 19 Jan 2023 15:52:47 GMT</pubDate>
            <atom:updated>2023-01-23T21:10:25.377Z</atom:updated>
            <content:encoded><![CDATA[<h3>A SOAPy business — extending SOAP with OnBeforeRequestAdvanced (and then some)</h3><p><em>This is an article about developing with the </em><a href="https://www.outsystems.com/"><em>OutSystems Platform</em></a><em>. If you are not yet developing with OutSystems, you should :).</em></p><h3>Preface</h3><p>As an energy supplier company, my employer connects to the national message hub that all energy companies connect to, for the exchange of messages pertaining to energy-related things like meter readings and supplier switches. The central organization governing this is a semi-public body and as such, loves baroque standards like SOAP/XML and gratuitous use of Java-only standards. This makes it a pain to properly support in OutSystems.</p><p>Originally (back in 2011–2013), we wrote an extension using the C# implementation of WSE3 (which has been obsolete for almost a decade) that consumed all the SOAP services. This worked but apart from it being a lot of opaque C# code, WSE3 only supports TLS 1.0, not the slightly more modern TLS 1.2, the latter which was mandated a couple of years ago, forcing us to use a proxy that reroutes and re-encodes the traffic adding TLS 1.2.</p><p>Since version 11 of the Platform, OutSystems has the <a href="https://success.outsystems.com/Documentation/11/Reference/OutSystems_APIs/SOAP_Extensibility_API">SOAP Extensibility API</a> for handling scenarios that are all but out-of-the-box SOAP. Given that maintaining the old code, and having to run an extra server with expensive software, costs us both a lot of time and money, we decided to consume the SOAP services natively in OutSystems, and use the Extensibility API to extend the behaviour where necessary. This included:</p><ul><li>Adding client certificates;</li><li>Modifying received XML content to satisfy WCF (or rather, perhaps, to fix broken Java implementations of <a href="https://en.wikipedia.org/wiki/WS-Security">WS Security</a>);</li><li>Logging the received and sent XML messages as-is, including any added security layers.</li></ul><p>In this article, I will describe how we approached the above, and how we finally managed to get everything working¹.</p><h3>Breaking it down</h3><p>We had to accomplish a number of feats, as we needed to consume various services that each had their own peculiarities. Basically the main two problems were successfully sending and receiving messages, and logging everything that was sent and received.</p><p>As for sending and receiving, we had to deal with three different groups of services, each with their own history, which meant that all three groups needed a different approach (though two of them were pretty similar in their reliance on some form of WS Security, while the third one only needed a client certificate added).</p><p>As functionality-wise these groups of services are quite diverse, and calling them “group 1” etc. is boring, I’ll refer to them with the internal names: TMR, P4 and Portal².</p><h3>OnBeforeRequestAdvanced</h3><p>Before commencing, one word on the “OnBeforeRequestAdvanced” of a SOAP service. To add it, you select “New OnBeforeRequest (Advanced)” from the dropdown of the SOAP service’s “On Before Request” property:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*bESN3sgBQZI7c-VCiGBnEQ.png" /><figcaption>This is how it’s done. It’s Advanced. No other flavours available.</figcaption></figure><p>This adds an OnBeforeRequestAdvanced action to the service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*h0SW-ieETFEvj7mcHhaZTQ.png" /></figure><p>When you are used to REST services, with their On Before Request and On After Response, you probably wonder why there’s no “normal” On Before Request (but only an Advanced one) and why there’s no On After Response. That is because, technically, SOAP works differently from REST, and you setup the handling of both the request and the response before you call a SOAP method. I personally would’ve chosen a different name in Service Studio instead of trying to mimic that of REST (which predates the SOAP extensibility by a whole Platform version, as it was available already in Platform 10), but here we are, and we have to live with this slight confusion.</p><h3>Adding a client certificate</h3><p>Adding a client certificate to a message is fairly easy, as <a href="https://success.outsystems.com/Documentation/11/Extensibility_and_Integration/SOAP/Consuming_SOAP_Web_Services/Use_Advanced_Extensibility/Authenticate_using_a_client_certificate">an example</a> is available from <a href="https://success.outsystems.com/Documentation/11/Extensibility_and_Integration/SOAP/Consuming_SOAP_Web_Services/Use_Advanced_Extensibility">the extensibility examples</a> provided by OutSystems on their website. As the Portal only needs a client certificate, this was the easiest part and the start of our implementation. I consumed one of the WSDLs (there’s more than two dozen), wrote a simple extension with the code from the example, added an OnBeforeRequestAdvanced to the SOAP service in which I called the extension action, and lo and behold — it worked! The final code:</p><pre>ISOAPClient client = SoapRequest.GetCurrentClient();<br><br>CustomBinding customBinding = new CustomBinding();<br>customBinding.Elements.Add(new CustomMessageEncodingBindingElement(&quot;text/xml&quot;, &quot;utf-8&quot;, MessageVersion.Soap11, soapLogger));<br><br>var httpsTransportBindingElement = new HttpsTransportBindingElement<br>{<br>  RequireClientCertificate = true<br>};<br>customBinding.Elements.Add(httpsTransportBindingElement);<br><br>client.Endpoint.Binding = customBinding;<br>client.ClientCredentials.ClientCertificate.Certificate = CertificateLoader.LoadClientCertificate(ssStoreLocationClient, ssStoreNameClient, ssSubjectNameClient);</pre><p>CertificateLoader is a class I wrote that uses a X509Store to search for and load a specific certificate, returned as a X509Certificate2.</p><p>Now that was easy, and by far the easiest part of the exercise. Next up was trying to modify the content of the SOAP message, and more specifically the security headers.</p><h3>Modifying a SOAP response</h3><p>As I wrote above, TMR and P4 make use of WS-Security, which is an old Java standard that was never really supported properly in .NET (or at least not as flexible). This means that when trying to consume the SOAP services directly with WCF (<a href="https://learn.microsoft.com/en-us/dotnet/framework/wcf/whats-wcf">Windows Communication Foundation</a>, the underpinning of the .NET communication capabilities), we get all kinds of nasty errors as WCF decides it doesn’t like what the SOAP services send (we were in luck that at least what WCF sends is accepted by the SOAP service).</p><p>WCF uses so called <a href="https://learn.microsoft.com/en-us/dotnet/framework/wcf/bindings-overview"><em>bindings</em></a><em> </em>to configure the details of a communication channel, each binding consisting of multiple binding <em>elements</em>, each element specifying one of three categories:</p><ul><li>Protocol;</li><li>Encoding;</li><li>Transport.</li></ul><p>If you can use a default binding (as was the case with the Portal as described above) WCF is pretty easy to use, but if you need anything more complex, it gets, at least for the uninitiated, terribly complex quickly, as you need to create a custom binding with custom elements, and you need a lot of boilerplate to do so.</p><p>When we needed to add a client certificate, we didn’t need a custom binding, but only needed to update an existing transport binding. However, in order to modify a response, we need to create a custom encoding binding. Without guidance this would be nigh impossible, but luckily for us, <a href="https://learn.microsoft.com/en-us/dotnet/framework/wcf/samples/custom-message-encoder-custom-text-encoder">Microsoft supplies example code</a>, and OutSystems <a href="https://success.outsystems.com/Documentation/11/Extensibility_and_Integration/SOAP/Consuming_SOAP_Web_Services/Use_Advanced_Extensibility/Use_a_different_character_encoding">has created an example</a> from that code. So that’s what I started with. As can be viewed from the examples, we need four levels of custom code, in the below order. Shown here are the class names, which we need to instantiate and use:</p><pre>CustomBinding -&gt;<br>   MessageEncodingBindingElement -&gt;<br>      MessageEncoderFactory -&gt;<br>         MessageEncoder</pre><h4>CustomBinding</h4><p>Starting with the CustomBinding, an instance of this class must be assigned to EndPoint.Binding field of the current SOAP client, after the various custom elements are assigned to it:</p><pre>ISOAPClient client = SoapRequest.GetCurrentClient();<br>CustomBinding customBinding = new CustomBinding();<br><br>// do custom stuff here<br><br>client.Endpoint.Binding = customBinding;</pre><h4>MessageEncodingBindingElement</h4><p>The next step is creating a new custom class that is derived from MessageEncodingBindingElement. Defined in System.ServiceModel, it needs MessageVersion and CreateMessageEncodingFactory to be overridden. MessageVersion has return type MessageVersion, which in our case returns the value Soap11. CreateMessageEncodingFactory is the most interesting one, as it returns a MessageEncoderFactory, which we also need to derive a custom class from (see below).</p><p>In the custom class’s constructor, I pass I pass the media type, encoding and message version which are needed later on:</p><pre>public class CustomMessageEncodingBindingElement : MessageEncodingBindingElement<br>{<br>  public override MessageVersion MessageVersion { get; set; }<br>  internal string mediaType;<br>  internal string encoding;<br><br>  public CustomMessageEncodingBindingElement(string mediaType, string encoding, MessageVersion messageVersion)<br>  {<br>      MessageVersion = messageVersion ?? throw new ArgumentNullException(&quot;msgVersion&quot;);<br>      this.mediaType = mediaType ?? throw new ArgumentNullException(&quot;mediaType&quot;);<br>      this.encoding = encoding ?? throw new ArgumentNullException(&quot;encoding&quot;);<br>  }<br><br>  public override MessageEncoderFactory CreateMessageEncoderFactory()<br>  {<br>    return new CustomMessageEncoderFactory(mediaType, encoding, MessageVersion);<br>  }<br><br>  // everything else is boilerplate<br>}</pre><p>Once our custom MessageEncodingBindingElement is done, we can use Elements.Add of our customBinding to add it to the custom binding:</p><pre>ISOAPClient client = SoapRequest.GetCurrentClient();<br>CustomBinding customBinding = new CustomBinding();<br><br>customBinding.Elements.Add(new CustomMessageEncodingBindingElement(&quot;text/xml&quot;, &quot;utf-8&quot;, MessageVersion.Soap11));<br><br>client.Endpoint.Binding = customBinding;</pre><h4>CustomMessageEncoderFactory</h4><p>Our custom MessageEncoderFactory needs to override Encoder and MessageVersion. MessageVersion should return the message version (the same as for MessageEncodingBindingElement), Encoder should return our custom MessageEncoder.</p><pre>public class CustomMessageEncoderFactory : MessageEncoderFactory<br>{<br>  public override MessageEncoder Encoder { get; }<br>  public override MessageVersion MessageVersion { get; }<br><br>  public string MediaType { get; }<br>  public string Encoding { get; }<br>  <br>  internal CustomMessageEncoderFactory(string mediaType, string encoding, MessageVersion version)<br>  {<br>    MessageVersion = version;<br>    MediaType = mediaType;<br>    Encoding = encoding;<br><br>    Encoder = new CustomMessageEncoder(this);<br>  }<br>}</pre><p>It is, for some reason, customary to pass the factory to the custom MessageEncoder’s constructor instead of passing version, media type etc. as separate parameters. I don’t know why this is.</p><h4>CustomMessageEncoder</h4><p>Now we’ve arrived at the heart of the matter. The custom MessageEncoder is doing the heavy lifting, all other modifications were mainly boilerplate. As a MessageEncoder, our custom encoder needs to override ContentType, MediaType, MessageVersion and two versions of ReadMessage and WriteMessage.</p><pre>public class CustomMessageEncoder : MessageEncoder<br>{<br>  public override string ContentType { get; }<br>  public override string MediaType { get; }<br>  public override MessageVersion MessageVersion { get; }<br>  <br>  private readonly CustomTextMessageEncoderFactory factory;<br>  private readonly XmlWriterSettings writerSettings;<br><br>  public CustomMessageEncoder(CustomTextMessageEncoderFactory factory)<br>  {<br>    this.factory = factory;<br>    MediaType = factory.MediaType;<br>    MessageVersion = factory.MessageVersion;<br>    <br>    writerSettings = new XmlWriterSettings<br>    {<br>      Encoding = Encoding.GetEncoding(factory.Encoding)<br>    };<br><br>    ContentType = $&quot;{this.factory.MediaType}; charset={writerSettings.Encoding.HeaderName}&quot;;<br>  }<br><br>  public override bool IsContentTypeSupported(string contentType)<br>  {<br>    return true;<br>  }</pre><p>The XmlWriteSettings we’ll need later in the two WriteMessage methods. It’s also used here to set the content type header text. It is also important we return true in IsContentTypeSupported, as otherwise our content type will be ignored.</p><p>Now onto ReadMessage. The <a href="https://success.outsystems.com/Documentation/11/Extensibility_and_Integration/SOAP/Consuming_SOAP_Web_Services/Use_Advanced_Extensibility/Use_a_different_character_encoding">OutSystems example</a> I linked to above shows the boilerplate needed to make it work. For our purpose, we can copy as-is the second version (with a Stream as first parameter), as I found it isn’t used by OutSystems SOAP, or at least not in my use case. The first version is therefore where our interest lies. The boilerplate code retrieves the contents of the message in a memory stream. Since I needed to alter the XML (removing and/or adding nodes and such), I used an XmlDocument and XPathNavigator to walk the entire XML message (using XPathNavigator’s MoveToFollowing). In each iteration I checked whether the element was of interest to me (using the LocalName and NamespaceURI properties) and if so, deleted it (using DeleteSelf) or modified it (using SetValue). I won’t show the actual code here, as it is very specific to my usecase, but below I give some dummy examples. Finally, after walking the entire XML message, we need to write it back again. For this, we need to create a new memory stream, write the XML document to that stream, and pass it to another overload of ReadMessage. There was one complication, which I had to work around: XmlDocument.Save adds a UTF-8-encoded <a href="https://en.wikipedia.org/wiki/Byte_order_mark">BOM</a> to the stream. I tried various ways to get rid of it, but in the end I decided to just skip it before passing the stream to ReadMessage.</p><pre>public override Message ReadMessage(ArraySegment&lt;byte&gt; buffer, BufferManager bufferManager, string contentType)<br>{<br>  byte[] msgContents = new byte[buffer.Count];<br>  Array.Copy(buffer.Array, buffer.Offset, msgContents, 0, msgContents.Length);<br>  bufferManager.ReturnBuffer(buffer.Array);<br>  <br>  MemoryStream ms = new MemoryStream(msgContents);<br>  XmlDocument doc = new XmlDocument();<br>  doc.Load(ms);<br>  <br>  XPathNavigator nav = doc.CreateNavigator();<br><br>  while (nav.MoveToFollowing(XPathNodeType.Element))<br>  {<br>    if(nav.LocalName == &quot;BadTag&quot;)<br>    {<br>      nav.DeleteSelf();<br>    }<br>    else if(nav.LocalName == &quot;AlsoBad&quot;)<br>    {<br>      nav.SetValue(&quot;myValue&quot;);<br>    }<br>  }<br><br>  doc.PreserveWhitespace = true;              // no formatting<br>  MemoryStream msOut = new MemoryStream();<br>  doc.Save(msOut);<br>  <br>  byte[] bom = new byte[3];<br>  msOut.Read(bom, 0, 3);<br>  msOut.Position = (bom[0] == 0xEF &amp;&amp; bom[1] == 0xBB &amp;&amp; bom[2] == 0xBF) ? 3 : 0;        // skip BOM if present<br>  <br>  return ReadMessage(msOut, int.MaxValue);<br>}</pre><p>Since I didn’t need to change the outgoing messages, I used the boilerplate code for both WriteMessage methods.</p><p>With the above modifications I was able to successfully send and receive SOAP messages for all three systems. Now I just needed a way to store the actual SOAP message sent and received.</p><h3>Logging SOAP messages</h3><p>The first question one needs to ask before trying something like this is, isn’t there something out-of-the-box? After all, it is possible to set the Logging Level of a SOAP API to Full in Service Center, doesn’t that already save the messages?</p><p>Well, unfortunately, no. First, the Integration Logging, like the other logging of the Platform, is only stored for a moderate period of time (typically two weeks). This is far too short a time to be useful for us — we need messages to be preserved for months. Secondly, the Integration Logging of SOAP messages is pretty useless for two reasons: one, it only logs the body, but any security headers are stripped, and two, the reply message is stored, shall we say, rather spartan. An example:</p><pre>Request Headers:<br><br>Request message:<br>&lt;s:Envelope xmlns:s=&quot;http://schemas.xmlsoap.org/soap/envelope/&quot;&gt;<br>  &lt;s:Header&gt;<br>    &lt;Action s:mustUnderstand=&quot;1&quot; xmlns=&quot;http://schemas.microsoft.com/ws/2005/05/addressing/none&quot;&gt;urn:MeasureCorrectionRequest&lt;/Action&gt;<br>  &lt;/s:Header&gt;<br>  &lt;s:Body xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xmlns:xsd=&quot;http://www.w3.org/2001/XMLSchema&quot;&gt;<br>    &lt;MeasureCorrectionRequestEnvelope xmlns=&quot;urn:nedu:edsn:data:measurecorrectionrequest:1:standard&quot;&gt;<br>      &lt;EDSNBusinessDocumentHeader&gt;<br>        &lt;CreationTimestamp&gt;2023-01-17T07:20:01.0000000+01:00&lt;/CreationTimestamp&gt;<br>        &lt;MessageID&gt;42012bc5-7bf7-4c5f-a9a2-aa2ef7554703&lt;/MessageID&gt;<br>        &lt;Destination&gt;<br>          &lt;Receiver&gt;<br>            &lt;ContactTypeIdentifier&gt;EDSN&lt;/ContactTypeIdentifier&gt;<br>            &lt;ReceiverID&gt;8712423010208&lt;/ReceiverID&gt;<br>          &lt;/Receiver&gt;<br>        &lt;/Destination&gt;<br>        &lt;Source&gt;<br>          &lt;ContactTypeIdentifier&gt;DDQ_O&lt;/ContactTypeIdentifier&gt;<br>          &lt;SenderID&gt;8717755560001&lt;/SenderID&gt;<br>        &lt;/Source&gt;<br>      &lt;/EDSNBusinessDocumentHeader&gt;<br>      &lt;Portaal_Content /&gt;<br>    &lt;/MeasureCorrectionRequestEnvelope&gt;<br>  &lt;/s:Body&gt;<br>&lt;/s:Envelope&gt;<br><br>Status Code: OK (200)<br><br>Response Headers:<br>X-Frame-Options: SAMEORIGIN<br>X-Content-Type-Options: nosniff<br>X-XSS-Protection: 1; mode=block<br>Strict-Transport-Security: max-age=31536000; includeSubDomains<br>Content-Security-Policy: default-src &#39;self&#39;; object-src &#39;none&#39;; style-src &#39;self&#39; &#39;unsafe-inline&#39;; upgrade-insecure-requests;<br>Referrer-Policy: strict-origin-when-cross-origin<br>Keep-Alive: timeout=10, max=100<br>Connection: Keep-Alive<br>Content-Length: 840<br>Cache-Control: no-cache, no-store, must-revalidate<br>Content-Type: text/xml;charset=utf-8<br>Date: Tue, 17 Jan 2023 06:20:01 GMT<br>Server: Apache-Coyote/1.1<br><br>Response message:<br>&lt;SOAP-ENV:Envelope xmlns:SOAP-ENV=&quot;http://schemas.xmlsoap.org/soap/envelope/&quot;&gt;<br>  &lt;SOAP-ENV:Header /&gt;<br>  &lt;SOAP-ENV:Body&gt;... stream ...&lt;/SOAP-ENV:Body&gt;<br>&lt;/SOAP-ENV:Envelope&gt;</pre><p>As you can see, the Body only contains “… stream …”, so all interesting information is not logged. Compare that to the actual messages:</p><pre>Request:<br><br>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;s:Envelope xmlns:s=&quot;http://schemas.xmlsoap.org/soap/envelope/&quot;&gt;<br>   &lt;s:Body xmlns:xsd=&quot;http://www.w3.org/2001/XMLSchema&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;&gt;<br>      &lt;MeasureCorrectionRequestEnvelope xmlns=&quot;urn:nedu:edsn:data:measurecorrectionrequest:1:standard&quot;&gt;<br>         &lt;EDSNBusinessDocumentHeader&gt;<br>            &lt;CreationTimestamp&gt;2023-01-17T07:20:01.0000000+01:00&lt;/CreationTimestamp&gt;<br>            &lt;MessageID&gt;42012bc5-7bf7-4c5f-a9a2-aa2ef7554703&lt;/MessageID&gt;<br>            &lt;Destination&gt;<br>               &lt;Receiver&gt;<br>                  &lt;ContactTypeIdentifier&gt;EDSN&lt;/ContactTypeIdentifier&gt;<br>                  &lt;ReceiverID&gt;8712423010208&lt;/ReceiverID&gt;<br>               &lt;/Receiver&gt;<br>            &lt;/Destination&gt;<br>            &lt;Source&gt;<br>               &lt;ContactTypeIdentifier&gt;DDQ_O&lt;/ContactTypeIdentifier&gt;<br>               &lt;SenderID&gt;8717755560001&lt;/SenderID&gt;<br>            &lt;/Source&gt;<br>         &lt;/EDSNBusinessDocumentHeader&gt;<br>         &lt;Portaal_Content /&gt;<br>      &lt;/MeasureCorrectionRequestEnvelope&gt;<br>   &lt;/s:Body&gt;<br>&lt;/s:Envelope&gt;<br><br>Response:<br><br>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;SOAP-ENV:Envelope xmlns:SOAP-ENV=&quot;http://schemas.xmlsoap.org/soap/envelope/&quot;&gt;<br>   &lt;SOAP-ENV:Body&gt;<br>      &lt;ns0:MeasureCorrectionResponseEnvelope xmlns:ns0=&quot;urn:nedu:edsn:data:measurecorrectionresponse:1:standard&quot;&gt;<br>         &lt;ns0:EDSNBusinessDocumentHeader&gt;<br>            &lt;ns0:CreationTimestamp&gt;2023-01-17T07:20:01.293+01:00&lt;/ns0:CreationTimestamp&gt;<br>            &lt;ns0:MessageID&gt;9fa186ff-9926-4b42-8902-40ad30e63331&lt;/ns0:MessageID&gt;<br>            &lt;ns0:Destination&gt;<br>               &lt;ns0:Receiver&gt;<br>                  &lt;ns0:ContactTypeIdentifier&gt;DDQ_O&lt;/ns0:ContactTypeIdentifier&gt;<br>                  &lt;ns0:ReceiverID&gt;8717755560001&lt;/ns0:ReceiverID&gt;<br>               &lt;/ns0:Receiver&gt;<br>            &lt;/ns0:Destination&gt;<br>            &lt;ns0:Source&gt;<br>               &lt;ns0:ContactTypeIdentifier&gt;EDSN&lt;/ns0:ContactTypeIdentifier&gt;<br>               &lt;ns0:SenderID&gt;8712423010208&lt;/ns0:SenderID&gt;<br>            &lt;/ns0:Source&gt;<br>         &lt;/ns0:EDSNBusinessDocumentHeader&gt;<br>         &lt;ns0:Portaal_Content /&gt;<br>      &lt;/ns0:MeasureCorrectionResponseEnvelope&gt;<br>   &lt;/SOAP-ENV:Body&gt;<br>&lt;/SOAP-ENV:Envelope&gt;</pre><p>And the above message doesn’t contain any security headers, but if they would, these would also not be visible in the Integration Logging. So it’s clear we need our own logging. But how to approach this?</p><h3>SoapLogger</h3><p>The problem consists of two main topics: retrieving the actual message content, and storing it in some accessible location. As for the latter, since one of the requirements is long retention of messages, using the General Log is out, as it has the same problems as the Integration Log (only about two weeks of retention), and it isn’t very well searchable. Ideally, the messages would be stored in the database, so we can easily search their metadata. Luckily for us, the Platform provides a way to do that.</p><p>To log information to an OutSystems Entity, OutSystems has, in the OutSystems.HubEdition.RuntimePlatform.Log namespace, an action available called CustomLog. This action allows one to write asynchronously, in an extension, an entity record to an OutSystems entity, which must be defined in a normal module. What we are interested in are the timestamp of the message (when was it sent/received), whether it was a request or response, the URL of the endpoint, the action, and of course the content. Hence, our datamodel looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*DE2T9tcYh87JioUOeF_bEw.png" /><figcaption>It’s a log of SOAP!</figcaption></figure><p>In our extension, I created a new class, SoapLogger, that has a constructor that takes two parameters, action and url, the use of which we will see later. It also has a method SetLogEntityRecord, which is called by an action defined in Integration Studio, which allows to set the entity record used for the logging (we’ll see below how it’s used). Then there are two actions for logging, LogMessage, which allows logging a SOAP message and is called by our CustomMessageEncoder (also explained below), and which calls the actual logging action, CustomLogEntityRecord. The entire class looks like this:</p><pre>public class SoapLogger<br>{<br>  private static Type entityRecordType;<br>  private static object entityRecord;<br>  private static int requestId;<br>  private static int responseId;<br>  private static bool isInitialized = false;<br><br>  private readonly string action;<br>  private readonly string url;<br><br>  public static void SetLogEntityRecord(object entityRecord, int requestId, int responseId)<br>  {<br>    SoapLogger.entityRecord = entityRecord;<br>    entityRecordType = entityRecord.GetType();<br>    SoapLogger.requestId = requestId;<br>    SoapLogger.responseId = responseId;<br>    <br>    isInitialized = true;<br>  }<br><br>  public SoapLogger(string action, string url)<br>  {<br>    this.action = action;<br>    this.url = url;<br>  }<br><br>  public void LogMessage(string message, bool isRequest)<br>  {<br>    CustomLogEntityRecord(isRequest ? requestId : responseId, DateTime.Now, url, action, message);<br>  }<br>  <br>  private void CustomLogEntityRecord(int typeId, DateTime instant, string url, string action, string contents)<br>  {<br>    if (!isInitialized) return;<br><br>    FieldInfo baseStruct = entityRecordType.GetFields()[0];<br>    object structValue = baseStruct.GetValue(entityRecord);<br>    Type structType = structValue.GetType();<br>    <br>    structType.GetProperty(&quot;ssType&quot;).SetValue(structValue, typeId, null);<br>    structType.GetProperty(&quot;ssDateTime&quot;).SetValue(structValue, instant, null);<br>    structType.GetProperty(&quot;ssURL&quot;).SetValue(structValue, url, null);<br>    structType.GetProperty(&quot;ssAction&quot;).SetValue(structValue, action, null);<br>    System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();<br>    structType.GetProperty(&quot;ssContent&quot;).SetValue(structValue, encoding.GetBytes(contents), null);<br>    <br>    baseStruct.SetValue(entityRecord, structValue);<br>    <br>    new CustomLog(entityRecord).Write();<br>  }<br>}</pre><p>In CustomLogEntityRecord, the attributes of the entity record are set (by their hardcoded name, prepended with “ss” as is the custom in Integration Studio) to the values passed as input parameters. CustomLog then asynchronously writes the record to the database.</p><p>Before we can use any of this, the module that calls a SOAP action must register the entity record. For this, I created the following action in the extension:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*yaHClyAjqTeBf_nTuNOdEw.png" /></figure><p>The code behind it is pretty simple:</p><pre>/// &lt;param name=&quot;ssLogEntityRecord&quot;&gt;Entity record of type SOAP_LOG used for logging&lt;/param&gt;<br>/// &lt;param name=&quot;ssLogTypeRequestId&quot;&gt;Id of SOAP_LOG_TYPE for SoapRequest&lt;/param&gt;<br>/// &lt;param name=&quot;ssLogTypeResponseId&quot;&gt;Id of SOAP_LOG_TYPE for SoapResponse&lt;/param&gt;<br>public void MssRegisterSoapLogger(object ssLogEntityRecord, int ssLogTypeRequestId, int ssLogTypeResponseId)<br>{<br>  SoapLogger.SetLogEntityRecord(ssLogEntityRecord, ssLogTypeRequestId, ssLogTypeResponseId);<br>} // MssRegisterSoapLoggerNew</pre><p>In our module, we call it like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*jsQvqqhr6M_0BLwrEXtPSw.png" /></figure><p>LogRecord is of type SOAP_LOG Record:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*sqKHiybECdNVJyqhz4nvMA.png" /></figure><p>What happens here, is that RegisterSoapLogger gets a record of type SOAP_LOG Record as parameter (as an object), which it stores, via SetLogEntityRecord, in the SoapLogger object for later use.</p><p>In order to actually use the SoapLogger, we need to create an instance of the class, and pass that to the constructor of our CustomMessageEncodingBindingElement (which we’ll have to extend for this purpose):</p><pre>SoapLogger soapLogger = new SoapLogger(SoapRequest.GetActionName(), SoapRequest.GetCurrentClient().Endpoint.Address.Uri.ToString());<br>...<br>customBinding.Elements.Add(new CustomMessageEncodingBindingElement(&quot;text/xml&quot;, &quot;utf-8&quot;, MessageVersion.Soap11, soapLogger);</pre><p>Next, we need to add the SoapLogger to CustomMessageEncodingBindingElement:</p><pre>public class CustomMessageEncodingBindingElement : MessageEncodingBindingElement<br>{<br>  ...<br><br>  internal SoapLogger soapLogger;<br><br>  public CustomMessageEncodingBindingElement(string mediaType, string encoding, MessageVersion messageVersion, SoapLogger soapLogger)<br>  {<br>      MessageVersion = messageVersion ?? throw new ArgumentNullException(&quot;msgVersion&quot;);<br>      this.mediaType = mediaType ?? throw new ArgumentNullException(&quot;mediaType&quot;);<br>      this.encoding = encoding ?? throw new ArgumentNullException(&quot;encoding&quot;);<br><br>      this.soapLogger = soapLogger;<br>  }<br><br>  public override MessageEncoderFactory CreateMessageEncoderFactory()<br>  {<br>    return new CustomMessageEncoderFactory(mediaType, encoding, MessageVersion, soapLogger);<br>  }<br><br>  ...<br>}</pre><p>Since we need the SoapLogger all the way down in our custom MessageEncoder, we first extend our custom MessageEncoderFactory:</p><pre>public class CustomMessageEncoderFactory : MessageEncoderFactory<br>{<br>  ...<br><br>  public SoapLogger SoapLogger { get; }<br>  <br>  internal CustomMessageEncoderFactory(string mediaType, string encoding, MessageVersion version)<br>  {<br>    ...<br><br>    SoapLogger = soapLogger;<br><br>    Encoder = new CustomMessageEncoder(this);<br>  }<br>}</pre><p>Finally, we can actually call the logging action from ReadMessage and WriteMessage of our custom MessageEncoder:</p><pre>public override Message ReadMessage(ArraySegment&lt;byte&gt; buffer, BufferManager bufferManager, string contentType)<br>{<br>  byte[] msgContents = new byte[buffer.Count];<br>  Array.Copy(buffer.Array, buffer.Offset, msgContents, 0, msgContents.Length);<br>  bufferManager.ReturnBuffer(buffer.Array);<br><br>  MemoryStream ms = new MemoryStream(msgContents);<br><br>  factory.SoapLogger.LogMessage(Encoding.UTF8.GetString(ms.ToArray()), false);<br><br>  ms.Position = 0;<br>  XmlDocument doc = new XmlDocument();<br>  doc.Load(ms);<br><br>  ...<br>}</pre><p>Note the code to set the stream’s position to 0, as the ms.ToArray() has moved it to the end.</p><p>For WriteMessage we do something similar:</p><pre>public override ArraySegment&lt;byte&gt; WriteMessage(Message message, int maxMessageSize, BufferManager bufferManager, int messageOffset)<br>{<br>  ...<br><br>  byte[] messageBytes = ms.GetBuffer();<br>  int messageLength = (int)ms.Position;<br><br>  factory.SoapLogger.LogMessage(Encoding.UTF8.GetString(ms.ToArray()), true);<br><br>  ms.Close();<br><br>  ...<br>}</pre><p>Since we close the stream after logging the message, there is no need to reset its position.</p><p>And that’s “all” there’s to it. Now whenever a SOAP action is called, we log the request message and the response message to the database. We have created a simple list screen to view and filter all messages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SiwISMcfDrmJpv1NIZytQg.png" /><figcaption>For some reason, the static entity SOAP_LOG_TYPE is translated, while the rest isn’t. It reads Request (“Aanvraag”) and Response (“Antwoord”).</figcaption></figure><p>It’s a pitty that most of this is very specific to one’s needs, so I can’t easily create an extension I can release in the Forge.</p><p>A note on security: since the messages are logged as-is, they may contain content like passwords that should not be logged at all. To mitigate this, one should scrub these from the XML, in a manner similar to what I’ve shown in the paragraph “Modifying a SOAP response”. I’ll leave it as an exercise for the reader to come up with a working solution 😁.</p><p>And a final note: a while ago we received a SOAP message that was so large, that we got an error from WCF. It turned out that we needed to increase the maximum message size, which by default is 64KiB (65535 bytes). This is done by setting the MaxReceivedMessageSize property of the same HttpsTransportBindingElement that’s also used to enable the client certificate by setting its RequireClientCertificate property to true.</p><pre>var httpsTransportBindingElement = new HttpsTransportBindingElement<br>{<br>  RequireClientCertificate = true,<br>  MaxReceivedMessageSize = 0x400000<br>};<br>customBinding.Elements.Add(httpsTransportBindingElement);</pre><p>¹I’m writing “we”, as in “my company”, but I was the only developer involved.</p><p>²The oldest one is TMR, relying on WS-Security and having a crude XML schema with Dutch abbreviations as tag names. It is used for requesting the historical measurement data of energy connections. We use these for validating and estimating new meter readings.</p><p>The second oldest is P4, also relying on WS-Security but having a more sensible naming scheme with descriptive English tag names. It is used for requesting data read from smart meters. Energy suppliers request this data by sending a request message to the central hub, which disperses it to the right recipient, which is a <a href="https://en.wikipedia.org/wiki/Transmission_system_operator">transmission system operator</a> (TSO) that administers the smart meters. The TSO collects the data by reading the smart meter via GPRS or some other technique, which can then be retrieved by polling for ready data.</p><p>The newest (since 2013) is the Portal, which is basically the access point to all other data exchange, including supplier changes, meter readings and contract data. It doesn’t use WS-Security, but only needs a client certificate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6512dcfbc8" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/a-soapy-business-extending-soap-with-onbeforerequestadvanced-and-then-some-b6512dcfbc8">A SOAPy business — extending SOAP with OnBeforeRequestAdvanced (and then some)</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Paginating, sorting and filtering data from a Data Action]]></title>
            <link>https://itnext.io/paginating-sorting-and-filtering-data-from-a-data-action-f5cb997fdc60?source=rss-f3f3765190c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5cb997fdc60</guid>
            <category><![CDATA[outsystems]]></category>
            <dc:creator><![CDATA[Kilian Hekhuis]]></dc:creator>
            <pubDate>Mon, 24 Jan 2022 15:19:18 GMT</pubDate>
            <atom:updated>2022-01-31T12:02:26.650Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This is an article about developing with the </em><a href="https://www.outsystems.com/"><em>OutSystems Platform</em></a><em>. If you are not yet developing with OutSystems, you should :).</em></p><h3>Preface</h3><p>In a Reactive app, Screen Aggregates can be used to retrieve data. If you display the data in a Table widget, and you want to paginate, sort and/or filter the data, you need to solve that in your Screen Aggregate, as described <a href="https://success.outsystems.com/Documentation/11/Developing_an_Application/Design_UI/Creating_and_editing_tables/Table_pagination_and_sorting">in the official documentation</a>. However, oftentimes data is coming from a Data Action that calls a REST Service Method or a Service Action, and those actions may not support pagination, sorting or filtering, but instead return a full, unpaginated, and possibly unsorted list of all records available. Unfortunately, OutSystems hasn’t described a canonical way of how to handle paging, sorting and/or filtering in such a case. In this article I will describe what in my opinion is the best, and fastest way to do this.</p><h3>Pagination</h3><p>Let’s first take a look at pagination. In a Reactive app, pagination is handled by the OutSystems UI <em>Pagination</em> widget. The Pagination widget has three important parameters, <em>StartIndex</em>, <em>MaxRecords </em>and <em>TotalCount</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*jZc8CXLhZbKaObtkggcyxQ.png" /><figcaption>The Pagination widget, part of the Navigation UI Flow.</figcaption></figure><p>StartIndex is the index of the first record of data that is displayed, MaxRecords the total number of records displayed, and TotalCount the total number of records of all pages combined. Using MaxRecords and TotalCount, the Pagination widget calculates the total number of pages, and using MaxRecords and StartIndex, the Pagination widget calculates the current page. It also uses these values to display the <em>1 to 10 of 100 items</em> message (where 1 is the value of StartIndex, 10 is the value of StartIndex + MaxRecords − 1 and 100 is the value of TotalCount).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f8xFxboPkNA8wuyIrhMdUA.png" /><figcaption>The Pagination widget in action. There’s a lot of records in this Table.</figcaption></figure><p>It is important to note that what the Pagination widget displays, is totally independent of what the Table widget displays. In Traditional Web apps, the Table Records widget holds the total list, and via some actions referenced from RichWidgets, the Table Records itself is responsible for displaying the right page, making it in sync with the RichWidgets List_Navigation widget (which is the Traditional Web equivalent of the OutSystems UI Pagination widget). But in Reactive apps, the Table widget displays whatever is in the list of its Source property. So it is up to the developer to make sure it displays the right content, in sync with the Pagination widget.</p><p>As I wrote in the preface of this article, if the Source is the output of a Screen Aggregate, the OutSystems documentation tells you in detail what to do. But on the matter of Data Actions, it is unfortunately silent. As might be clear by now, since the Table widget displays whatever is in the source list, you need to make sure the source list contains <em>only</em> the records of a single page, the page being determined by the Pagination widget. If you receive from your Data Action a full list of records, you will need to take a subsection from that list, and use that as the source list.</p><p>In order to do so, you need of course a Data Action (in the context menu of the Screen called “Fetch Data from Other Sources”). Inside the Data Action, you call the REST method, Server Action, Service Action or SQL query or even Aggregate that provides the data. Of course, when using a SQL or Aggregate you could paginate using the prescribed method, but especially with heavy queries that do not return too many records (let’s say a few thousand max), you may be better off fetching the data once, instead of refreshing the query when the user skips to another page (or sorts or filters the data, see further on in this article). The Data Action should output the entire list that is fetched — this becomes your source list. Do not use this list as Source for the Table though! As I mentioned above, this would display all records in the Table, and no pagination will be in place.</p><p>When you drag a Pagination widget onto the canvas, its property pane shows red borders for the mandatory input parameters, and also for the Handler Screen Action of the OnNavigate event. This event will be fired whenever the user clicks on one of the navigation buttons (backward, forward, page number) or enters a page number (in case the ShowGoToPage property is set to True).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*hy2ZQCbncVSNkQSovAEJ1Q.png" /><figcaption>Pagination property pane</figcaption></figure><p>If you click on the Handler’s down arrow, and select <em>New Client Action</em>, a client action will be created called PaginationOnNavigate, which has a single input parameter called NewStartIndex.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/1*DnVZIrimVujtjhjhSZNn7g.png" /><figcaption>Auto-generated PaginationOnNavigate</figcaption></figure><p>After you have created the PaginationOnNavigate, set the Data Action’s “On After Fetch” property to PaginationOnNavigate, with a NewStartIndex of 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*H-LCgCl-BB_zaTtOE43QRA.png" /><figcaption>The On After Fetch set to PaginationOnNavigate.</figcaption></figure><p>This ensures that whatever we will do to make sure the right data is displayed, it will also happen when the data has just been fetched.</p><p>Since the Table cannot directly use the Data Action’s output as Source, we need a Screen Local Variable to hold the data that we actually want to display. For the sake of simplicity, I’ll assume here that the data type of that list is the same as the Data Action’s output, but depending on whether you want to perform any calculations or look-ups based on the data, you may use a different structure. For our example, let’s call the Local Variable <em>PagedData</em>. Next, set PagedData as Source for the Table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*CUfRDukIbZrmVgC-GzmIQQ.png" /><figcaption>Our PagedData list is now Source of our Table</figcaption></figure><p>The Pagination Widget still has errors, as we didn’t yet set the StartIndex, MaxRecords and TotalCount properties. We will create three Screen Local Variables for this, for simplicity’s sake we’ll call them <em>StartIndex</em>, <em>MaxRecords</em> and <em>TotalCount</em>. Assign these to the Pagination’s properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*SAYNjTxsjo2-NY4dwDpmEA.png" /><figcaption>The Pagination widget properties all filled in</figcaption></figure><p>To prevent the Pagination to show an error the moment the Screen is opened, set the MaxRecords default value to whatever number of records you want to display per page, e.g. 10.</p><p>In the PaginationOnNavigate action, we now need to populate the PagedData list with the right records from the Data Action’s output. Since the Platform does not provide us with an out-of-the-box action to take a slice from a list, we need to use a For Each to do so. We iterate over the Data Action’s output, using a Start Index equal to the NewStartIndex input variable, and a Maximum Iterations equal to MaxRecords. Add a ListAppend inside the For Each, that appends the Current of the Data Action’s output list to the PagedData list. Since PaginationOnNavigate will be called multiple times, we also need to ListClear PagedData before the For Each, or else the list would grow with each call.</p><p>One last thing we need to do is set the Screen Local Variable StartIndex to the NewStartIndex input variable, and the TotalCount to the Length of the Data Action’s output list, so the Pagination widget has the right values to work with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*ok0QxSz-OnwLgkM47e-iXA.png" /><figcaption>The finished PaginationOnNavigate. For clarity the assignments have been split, but you’d typically put them in a single assign.</figcaption></figure><p>With this, our pagination is finished!</p><h3>Sorting</h3><p>Next let’s take a look at sorting. The Table widget has support for sorting out of the box. If you click a Header Cell, you will see one of its properties is called <em>Sort Attribute</em>. If it is set, the column automatically gets a double arrow after the column name, which you can click to sort the column. By default, when you drag a variable onto the Table, Service Studio will automatically create sorting for all columns. Remove the Sort Attribute text for columns you do not want sorting for.</p><p>Another property of a Table is the On Sort event, that fires when the user clicks the header of a sortable column. To generate a Screen Action for sorting, click the down arrow and select “New On Sort Client Action”. Service Studio will then create an action called OnSort, with an argument SortBy. In the Table properties, SortBy is set to <em>ClickedColumn</em>. This is the output of the On Sort event that the Table generates when you click a sortable header. It is set to whatever text you have set for the column’s Sort Attribute. Service Studio also generates a Screen Local Variable called <em>TableSort</em>. This variable is used inside the auto-generated OnSort action, and will hold the Sort Attribute value of the currently sorted column.</p><p>If you open the OnSort action, you’ll see the following auto-generated code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*Db-v-v4Vz-Nif98Gsba2hQ.png" /><figcaption>Auto-generated sorting code. Yes, that first If has only four grid lines above it, while everything else has five 😠.</figcaption></figure><p>We can’t use this code directly, if only because it has a Refresh Data, which we won’t use. But let’s analyze what the code does exactly. In the If, the code checks whether we have a change in the column that’s clicked. To this end, it compares the TableSort local variable with the SortBy input. Since TableSort holds the value of the Sort Attribute property of the currently sorted column (but see below), when TableSort is equal to SortBy, there’s no change. I’m not sure why it checks for SortBy being empty — as far as I know, SortBy is always filled. When there is a change, the assignment in the True branch adds “DESC” (with a leading space) to the TableSort variable. This does two things: it makes sure that if you click the column again, the If goes to the False branch (so it toggles between ascending and descending sorting), and secondly, if you have a Screen Aggregate it should use TableSort as Dynamic Sort, and by adding DESC it will sort descending instead of ascending. The False branch simple sets TableSort to the SortBy input, and both branches then set the StartIndex to 0, so our Pagination widget knows to start on page 1. Note that it here says StartIndex2, which seems a bug in Service Studio¹: it created another StartIndex variable, while we already created one ourselves. Finally, it calls Refresh Data to refresh the Screen Aggregate your Table is presumably bound to.</p><p>As I said, we can’t use this code, but we will create something similar to sort our in-memory Data Action output list. For start, adding “DESC” to our column name is of no use, as we don’t use an Aggregate. Therefore we’ll add another Screen Local Variable, called IsSortDesc. If it’s True, we’ll sort descending, otherwise we’ll sort ascending. Like in the auto-generated code, we first check whether we clicked the same column by comparing TableSort with SortBy. If they’re equal, we toggle the value of IsSortDesc (by assigning “not IsSortDesc” to IsSortDesc²). Otherwise, we’ll assign SortBy to TableSort, and set IsSortDesc to False, since we always want to start by sorting ascending³.</p><p>For sorting the list, we use ListSort. The list we’ll sort is the output of the Data Action (<em>not</em> the PagedData of course, we want the page to show one page of the total sorted data, not just sort whatever is on the page). The “By” property must be set based on the value of TableSort/SortBy (they’re equal because of the assign) using an If in the expression. The Ascending property can be set to “not IsSortDesc”⁴.</p><p>Next, we call PaginationOnNavigate, using a NewStartIndex of 0. Since PaginationOnNavigate uses the output of the Data Action to fill the PagedData list, it will now use the sorted list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*pjARRwDcNkAPjMYAuMqv8g.png" /><figcaption>Our own OnSort sorting action. Look how nicely evenly spaced everything is! 🥰 Note that “Name” and “Function” are two attributes of the DataAction1.Output list structure in this example.</figcaption></figure><p>One final thing to do is to give TableSort a default value that’s equal to whatever column is sorted on, if the Data Action returns a sorted list (typically the REST method or Service Action will return the list sorted by some attribute, alternatively you could use ListSort in the Data Action to sort it after retrieval). If you don’t do this, the first time the user clicks that column header nothing will happen, as TableSort will not be equal to SortBy, and ListSort will sort ascending.</p><p>And that’s it for sorting!</p><h3>Filtering</h3><p>The last thing we typically want to do, is to allow the user to filter the list. I’m going to assume you have some filter input fields in a form, a “Filter” button to filter on that input and a “Clear” button to clear the input. You may have a different implementation, that’s not important, as long as there are some values to filter the list on.</p><p>The first thing to realize is that, since we have no Aggregate to do the filtering, we need ListFilter to filter our list. A filtered list is most of the time a subset of a total list, so in our case a subset of the Data Action’s output. It’s this subset that needs to be paginated and sorted, so both sorting and pagination need to be applied to the subset. To accomplish this, we first create another Screen Local Variable to hold this filtered list, of the same type as the Data Action’s output. Let’s call it FilteredData.</p><p>Next, in the PaginationOnNavigate action, we replace the For Each Record List input with FilteredData (from DataAction.Output), and the ListAppend element with FilteredData.Current. Also, the assignment to the TotalCount is changed so that it takes the length of the FilteredData list instead of the Data Action’s output list⁵.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*D0MNcPVyOeuWgu2IP4M56g.png" /><figcaption>Changes to the PaginationOnNavigate action</figcaption></figure><p>Next, we need to be able to actually filter the list. To this end, we create a new Screen Action called OnFilter. We also set the “Filter” button’s On Click event to the new OnFilter action. Next we need to write the logic of OnFilter. First, we’ll assign the output of the Data Action to FilteredList — if we don’t filter at all (because our filter inputs are empty), this is the default situation. Next, we’ll check for each filter input on our screen whether the user has input something. For this example, I’ll assume filter inputs for a “Name” column and a “Function” column. I also use a Local Screen Variable of a structure type called <em>Filter</em> for the filter inputs, but you can use seperate variables if you like.</p><p>If a filter input is filled, we apply a ListFilter to the filtered list with whatever condition you want to use for filtering. In the example I have free text inputs, and I want to filter the list on occurences of that text, comparable to a “like” in an Aggregate. To do this, we use a condition that uses the built-in Index function, searching for the filter input in the pertaining list attribute, converting both to lowercase (as we want to compare case-insensitive, like in a database search). After this, we assign the output of the ListFilter to FilteredData. We repeat this for each filter input⁶.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*-k3VxMHNHuPlAT7B-Lprsg.png" /><figcaption>What we’ve done so far. But this is not all!</figcaption></figure><p>The next thing to do is sort the filtered list. Of course, if the Data Action’s output list was already filtered, the sorted list will also be filtered. But if, after filtering, the user sorts the list differently, we need to sort the filtered list, and not the Data Action’s output⁷. To do this, we call the OnSort action after filtering. However, as you may recall, the OnSort starts with checking whether the TableSort is equal to Sort, and if so, reverts the sorting order. In the case of applying a Filter this shouldn’t be done of course, as every time the user filters the list, the sorting order would reverse. Therefore we add a non-mandatory boolean parameter to the OnSort action, called IsFromFilter, that indicates whether it’s called from the OnFilter action (True) or from the Table’s OnSort event (False).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*oEBSgi9ZbDFOW524Hv9vVQ.png" /><figcaption>OnSort with IsFromFilter added.</figcaption></figure><p>We then call OnSort from OnFilter, with IsFromFilter set to True. Note that since OnSort already calls PaginationOnNavigate, we needn’t call that action from OnFilter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*CDGtW1jk7y_4vx5Be1YKuA.png" /><figcaption>The final version of OnFilter, with OnSort added.</figcaption></figure><p>Now we need to handle the “Clear” button. When pressed, all filters should be cleared, and the data in the Table should represent that. You could go about this in two ways: add a seperate Screen Action called OnFilterClear or the like, in which you clear the filter in an assignment and call OnSort. Or, and that’s what I typically opt for, add a non-mandatory <em>IsClear</em> input parameter to the OnFilter, set it to True in the “Clear” button’s On Click event (after assigning OnFilter to it), and handle it in OnFilter. Since it’s all rather trivial, I show it below as-is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*4i-7Agos-96XriYOv-k3gw.png" /><figcaption>The OnFilter action including clearing the filter. The “Dummy” assignment is there just for the arrows to flow nicely.</figcaption></figure><p>The last-but-one thing to do, is update OnSort. First, we need to use the new IsFromFilter input parameter. If its value is True, we bypass the code that checks for the column to sort on, and go directly to the sorting itself. Also, since we should now use the FilteredData list instead of the Data Action’s output, ListSort must use the filtered list instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*e4p-k1QWZ4SEOwrHitKgtA.png" /><figcaption>OnSort with added check on IsFromFilter and a modified ListSort.</figcaption></figure><p>Finally, because both the PaginationOnNavigate and OnSort now use FilteredData instead of the Data Action’s output list, we need to assign the output list to FilteredData directly after the Data Action is finished. Currently, we call PaginationOnNavigate as On After Fetch action, but we cannot assign it there, as that would reset the filtered list to the full list, and void any attempt to filter the list. Instead, we should call OnFilter, since the assignment is already in there. As an added bonus, if we have a default filter that’s not empty, the list will be filtered according to that filter (and subsequently sorted correctly), and shown filtered to the user without them having to press the “Filter” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*81u-ba7LvEbUnBhS-beeYA.png" /><figcaption>On After Fetch now calls OnFilter</figcaption></figure><p>And that’s it, we now have filtering in place as well!</p><h3>Closing words</h3><p>What seems like a fairly trivial thing to do when you have a Screen Aggregate to fetch your data, viz. pagination, sorting and filtering, is slightly less trivial when the data comes from a Data Action that returns a non-paginated list (in my experience a common scenario). In this article I’ve shown how to accomplish that, fully client-side, by using ListAppend, ListSort and ListFilter — I hope it will be helpful. For reference, I also <a href="https://www.outsystems.com/forge/component-overview/12354/paginating-data-action-example">uploaded a Forge Component</a> that shows the above in action, using a dummy REST service to retrieve the data. And as always, happy coding!</p><p>¹ Or, likely, it is deliberate as Service Studio can’t know whether the StartIndex we already had is used for the current Table, as there is no connection between the Table and the Pagination widget.</p><p>²I’ve seen code like “If(IsSortDesc = True, False, True)”, but please for the love of god don’t ever do this!</p><p>³Of course, if you really must, you could keep track of the last sort order for each column, and use that, but I wouldn’t recommend it as it may confuse the user.</p><p>⁴Alternatively you could use a variable called IsSortAsc, set its default value to True, assign it to True if TableSort is not equal to SortBy, and use it directly as input to Ascending.</p><p>⁵There’s more than one location we could assign the TotalCount. We originally did it in the PaginationOnNavigateAction because it was called after the Data Action had finished (as the Data Action’s On After Fetch action). Although we will change this, it’s still a convenient location, since we then only need the assignment once.</p><p>⁶It is possible to use a single ListFilter, combining the conditions for all filters. However, the code becomes pretty illegible pretty quickly once you have more than a filter or two, so I always opt for seperate ListFilters. In my experience, this doesn’t lead to performance or memory problems, but if you’d have a really large list, combining the filters may be necessary for performance. But as always, don’t optimize prematurely!</p><p>⁷Alternatively, we could sort the output list, then filter it every time the user applies a different sorting, but that would be quite wasteful, compared to sorting only the filtered list without subsequent filtering of the output list again.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5cb997fdc60" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/paginating-sorting-and-filtering-data-from-a-data-action-f5cb997fdc60">Paginating, sorting and filtering data from a Data Action</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exposing an OutSystems REST service with an OAuth-style authorization]]></title>
            <link>https://itnext.io/exposing-an-outsystems-rest-service-with-an-oauth-style-authorization-fded258cbe14?source=rss-f3f3765190c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/fded258cbe14</guid>
            <category><![CDATA[json-web-token]]></category>
            <category><![CDATA[outsystems]]></category>
            <category><![CDATA[oauth2]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Kilian Hekhuis]]></dc:creator>
            <pubDate>Tue, 28 Sep 2021 13:02:09 GMT</pubDate>
            <atom:updated>2023-09-26T12:23:57.475Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This is an article about developing with the </em><a href="https://www.outsystems.com/"><em>OutSystems Platform</em></a><em>. If you are not yet developing with OutSystems, you should :).</em></p><h3>Preface</h3><p>When exposing a REST service using the OutSystems Platform (from hereon simply “OutSystems”), there’s no security out of the box: anyone with access to the end-point can call the service. OutSystems allows for setting a Username and Password for Basic Authentication, but that’s global, i.e. everyone calling the REST service must use that Username and Password. Obviously, that’s not quite flexible and secure, so I doubt this option is used often, other than perhaps for internal use between applications.</p><p>What any modern REST service that is going to be consumed by multiple users (or applications) must have is per-user authentication, so that the REST method called can verify the user is a valid user (and otherwise deny access by returning a <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403">403</a>), and return results (or allow updates) according to the user’s privileges.</p><p>The modern industry standard for authentication is <a href="https://oauth.net/2/">OAuth 2.0</a>, as described in <a href="https://datatracker.ietf.org/doc/html/rfc6749">RFC 6749</a>. This standard describes a flow of authentication and authorization, involving a <em>Resource Owner</em> (the one exposing the REST methods) and an <em>Authorization Server</em> (the one providing the authorization). Like all modern industry standards it’s fairly complex, but the basic gist is that the Resource Owner checks your credentials and if correct provides an <em>Authorization Grant</em> that can be used with the Authorization Server to provide you with an <em>Access Token</em>. The Access Token is then used as a proof of authorization when you make subsequent calls to the Resource Owner (i.e. when calling the REST methods for retrieving or storing information).</p><p>When I was searching for a way to properly secure our REST service between our customer portal and our back-end server (originally it was based on just exchanging a username/password in the header), I looked at OAuth 2.0, but found it too complex to fully implement. However, its basic premises appealed to me, so I set out to implement something similar myself, without adhering fully to OAuth 2.0 as described in the above mentioned RFC.</p><h3>JSON Web Tokens</h3><p>OAuth 2.0 <a href="https://datatracker.ietf.org/doc/html/rfc6750">describes the use</a> of a so-called <em>Bearer Token</em>, which is the technical way of supplying the above-mentioned Access Token to the Resource Owner. However, OAuth 2.0 does not describe the exact format of the Access/Bearer Token. In comes the <a href="https://jwt.io/">JSON Web Token</a>, or JWT for short. A JWT is one format of Access Token (defined in <a href="https://datatracker.ietf.org/doc/html/rfc7519">RFC 7519</a>), but one that comes with a lot of handy functionality. It is:</p><ul><li>Self-contained: all information the Resource Provider needs to check the validity of the token is embedded in the token itself;</li><li>Secure: the token is signed with the Authorization Server’s certificate. It can be decrypted with the public key (so everyone that has access to the JWT can read its contents), but cannot be tempered with using a man-in-the-middle attack, as it cannot be encrypted again.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*eeeO936i7XECDRzV.png" /><figcaption>The JWT logo. For once, a logo without the Google colours (I’m looking at you, Slack).</figcaption></figure><p>A JWT contains three different parts: a <em>header</em>, a <em>payload </em>and a <em>signature</em>. As the name implies, information is stored using JSON. The full JWT has the three parts Base64 encoded, each part separate by a dot (“.”).</p><p>The <strong>header </strong>is a so-called JOSE header (JSON Object Signing and Encryption, defined in <a href="https://datatracker.ietf.org/doc/html/rfc7515">RFC 7515</a>) and contains information about the signing and encryption used. It has the following attributes:</p><ul><li>typ — the “media type”, always JWT;</li><li>alg — the encryption algorithm used. It can be either RS256, which is RSA Digital Signature Algorithm with SHA-256 (asymmetric) or HS256, which is HMAC SHA-256 (symmetric)¹. RS256 is the way to go for an API, since you do not want to share a secret encryption key with whoever consumes your API;</li><li>kid — the key identifier, an optional attribute that indicates what key is used for the encryption. It has no predefined structure. I used this attribute to store the certificate’s thumbprint, so that I could verify that the right certificate was used to create it. I later learned that the x5t attribute is available for exactly that, but I didn’t bother changing it.</li></ul><p>There are more attributes possible, but these are the three I used.</p><p>The <strong>payload </strong>is the most interesting part. It contains so-called <em>claims</em>, which are (and I quote jwt.io) “statements about an entity (typically, the user) and additional data”. There are three types of claims:</p><ul><li>Registered claims — optional, predefined claims like issuer, expiration time, subject and audience (see below);</li><li>Public claims — claims that should either come from a <a href="https://www.iana.org/assignments/jwt/jwt.xhtml">predefined list</a> or be defined as a URI. I did not use any public claims;</li><li>Private claims — any custom claims that are agreed upon by two parties exchanging the data. I did not use any private claims.</li></ul><p>I only used registered claims, as I didn’t have a need for any others. The ones I used are:</p><ul><li>iss — issuer, the name of the issuing party. This could be your company name, or the name of the app, etc. I used a combination of company name and application;</li><li>sub — subject, identifies the resource the other claims are about. Typically this is a user identifier. In my case, I used a combination of a company Id and a user Id, as I could use these values in the validation of the JWT;</li><li>aud — audience, identifies the recipient of the JWT, e.g. the REST service or application you are targeting. In my case, I used the application code and my company’s domain name to form a pseudo-URL, but anything goes;</li><li>exp — expiration time, the UTC date/time when the token expires, specified as seconds since 1970–01–01 00:00:00 (the UNIX epoch). After this time, the token is no longer valid and must be rejected. Since the Platform doesn’t handle UTC well (at least not server-side), I just went for local time, as I didn’t need to cross time zones;</li><li>iat — issued at, the UTC date/time when the token was issued, specified as seconds since 1970–01–01 00:00:00. Again I encoded local time here instead of UTC.</li></ul><p>You may forgo on some of these claims, but I found them useful for verifying the token, as well as making sure that my single token provider could be used with multiple applications, without a token being valid for all applications. Also, for security reasons, it’s a good idea to have tokens that are only valid for a limited amount of time.</p><p>The last part of a JWT is the <strong>signature</strong>. The signature takes the first two parts, Base64 encoded with a separating dot, and encrypts the resulting string using the encryption algorythm. The result of the encryption is also Base64 encoded and added, again with a dot as separator, to the first two parts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*gcU8USKEcnPSyabg5cRbaw.png" /><figcaption>Example of a JWT and its parts (from <a href="https://jwt.io/introduction">jwt.io)</a>.</figcaption></figure><h3>Authentication flow</h3><p>As described above, there are three parties involved in authentication: the <em>user</em> (which may be an app), the <em>resource owner</em> and the <em>authorization server</em>. First, the user needs to login, by providing their username and password. These are sent to the authorization server that validates the username and password, and returns an <em>authorization grant</em>, or more commonly called <em>access token</em>, which in our case is a JWT. With every call to the resource owner, the access token is sent along, but not the username/password. The resource owner is responsible for validating the access token, and if valid, perform whatever action was requested (e.g. a data update or retrieval).</p><p>The following flow chart shows this procedure (modelled after the one from <a href="https://auth0.com/docs/authorization/flows/resource-owner-password-flow">auth0.com</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*n0GaXcFSJdB5ksyu7S8ZtA.png" /><figcaption>Authentication flow represented visually</figcaption></figure><ol><li>The user types their username/password on a login screen;</li><li>The web app sends the username/password to the authorization server;</li><li>The authorization server validates the username/password and if valid sends an access token;</li><li>The web app requests data from the resource owner sending the access token along for authorization;</li><li>The resource owner validates the access token and if valid sends the data.</li></ol><h3>Implementing the JWT</h3><p>There is <a href="https://www.outsystems.com/forge/component-overview/1853/jwt">at least one JWT component</a> in the Forge, but that seemed geared towards consuming external services, and not providing an authorization service yourself. Therefore I decided to write everything from scratch.</p><p>First, I created an extension for creating an verifying a JWT. This extension has the following actions:</p><ul><li>CertificateGet — retrieves a certificate from the Windows certificate store² based on the certificate subject, the store name and the store location, and returns the certificate (as an Object), the subject’s full name and the certificate’s thumbprint.</li><li>JwtCreate — creates a JWT based on a JOSE header, JWT claims and a certificate. It does this by Base64-encoding the header and claims, and encrypting the result with the certificate, then concatenating the encrypted part (also Base64-encoded). The only tricky part here was to cut the input in pieces, as the encryption works on a maximum block size.</li><li>JwtGetParts — retrieves and decodes (from Base64) the JOSE header and the JWT claims, and returns them (as JSON). This could also be done in OutSystems using the BinaryData extension, but I found it easier to do in an extension.</li><li>JwtVerify — verifies whether a given input is a valid JWT by decripting the signature with the provided certificate and comparing it against the header and claims.</li></ul><p>Next, I created a module that contained the actual actions used by both the authorization server and the resource owner. These actions are:</p><ul><li>JsonWebTokenGet — Used by the authorization server, it returns a JWT based on the provided JOSE header, JWT claims and certificate. The expiration timestamp is also an input parameter; the action calculates the value of the expiration time (which is the number of seconds since 1–1–1970) of the JWT claims if it is provided (otherwise it assumes the expiration time has already been set). It uses the CertificateGet and JwtCreate actions from the extension.</li><li>JsonWebTokenVerify — Used by the resource owner, it verifies a JWT by calling CertificateGet, JwtGetParts and JwtVerify from the extension, and outputs the JOSE header and JWT claims.</li></ul><h3>Implementing the authorization server</h3><p>Implementing the authorization server was fairly straightforward. I exposed a REST service with a TokenGet POST method (relative path /authentication). It has the following parameters:</p><ul><li>Authorization — a header parameter that should be set to a valid basic authentication string³.</li><li>Application — since we use the same authorization server for multiple applications, an application code is used to avoid a token allowing access to all applications. It is a header parameter encoded as X-Application in the HTTP header.</li></ul><p>If the username and password are valid for the given applications, a JWT is returned. The following claims are included in the JWT claims:</p><ul><li>Issuer — a fixed string that identifies our backend server;</li><li>Audience — a code that identifies the application;</li><li>Subject — as described above, a combination of a company Id and a user Id;</li><li>Expiration time — calculated with a fixed duration (configured via a site property), as we have no need for a dynamic duration;</li><li>Issued at — the time at which the token was issued.</li></ul><h3>Sending the token</h3><p>When the user logs in in our customer portal, the username and password are sent to the authentication server as described above. When the access token is returned, it is stored in a session variable (our customer portal still uses traditional web, as it is several years old). The module that handles the login exposes an action that returns the token from this session variable.</p><p>When the customer portal needs to requests data from the back-end server, it uses bearer authentication. Like basic authentication, it uses an Authorization header parameter³, but instead of “Basic” is the word “Bearer”, followed by a space and the JWT. We decided to add the Authorization header parameter to all methods, though in retrospect, we might’ve added it to the header in an On Before Request.</p><h3>Token verification</h3><p>When the back-end receives a request from the customer portal, it starts with validating the JWT, by calling the aforementioned JsonWebTokenVerify action. It then performs a validity check on the data contained in the token, by verifying:</p><ul><li>The issuer claim — the issuer must be equal to the issuer issued by the authorization server;</li><li>The audience claim — the application must be equal to the application that is handling the request;</li><li>The subject claim — the customer Id and user Id must be valid, and the user must have access to the company’s data;</li><li>The expiration time — if the token is expired, the request is denied⁴.</li></ul><p>If everything checks out, the back-end returns the requested data.</p><h3>Wrapping it up</h3><p>Though OAuth and JWT are complex topics, I have successfully secured our customer portal communication with an OAuth-style authentication scheme using JWTs. I hope it will inspire you to use a similar authentication when exposing REST services of your own.</p><h3>Where’s the code?</h3><p>I have deliberately not put any of the code I wrote in the Forge, for two reasons:</p><ol><li>I initially planned of integrating some of the code with the existing JWT library (which didn’t pan out in the end);</li><li>My code is geared towards our customer portal, and a Forge component would need a more flexible solution, e.g. with regards to the checks that are made.</li></ol><p>However, since the extension is the most difficult code to write, I’ll share here some of the C# code I wrote.</p><h4>Retrieving a certificate from the certificate store</h4><p>Not JWT-specific, but useful:</p><pre>public void MssCertificateGet(string ssSubjectNamePart, string ssStoreName, string ssStoreLocation, out bool ssIsFound, out object ssCertificate, out string ssSubjectNameFull, out string ssThumbprint)<br>        {<br>            ssIsFound = false;<br>            ssCertificate = null;<br>            ssSubjectNameFull = &quot;&quot;;<br>            ssThumbprint = &quot;&quot;;</pre><pre>var storeName = ssStoreName.Equals(&quot;M&quot;) ? StoreName.My : StoreName.TrustedPeople;<br>            var storeLocation = ssStoreLocation.Equals(&quot;C&quot;) ? StoreLocation.CurrentUser : StoreLocation.LocalMachine;</pre><pre>X509Store store = new X509Store(storeName, storeLocation);<br>            store.Open(OpenFlags.ReadOnly | OpenFlags.OpenExistingOnly);</pre><pre>X509Certificate2Collection collection = store.Certificates.Find(X509FindType.FindBySubjectName, ssSubjectNamePart, true);</pre><pre>if (collection.Count == 1)<br>            {<br>                ssIsFound = true;<br>                ssCertificate = collection[0];<br>                ssSubjectNameFull = collection[0].Subject;<br>                ssThumbprint = collection[0].Thumbprint;<br>            }<br>        } // MssCertificateGet</pre><h4>Creating a JWT</h4><pre>public void MssJwtCreate(string ssJoseHeader, string ssJwtClaims, object ssCertificate, out string ssJwt)<br>        {<br>            var cryptoProvider = ((X509Certificate2)ssCertificate).PrivateKey as RSACryptoServiceProvider;<br>            if (cryptoProvider == null)<br>            {<br>                throw new Exception(&quot;Cannot access private key&quot;);<br>            }</pre><pre>AsymmetricCipherKeyPair kp = DotNetUtilities.GetKeyPair(cryptoProvider);</pre><pre>var encryptEngine = new Pkcs1Encoding(new RsaEngine());<br>            encryptEngine.Init(true, (RsaKeyParameters)kp.Private);</pre><pre>string textToEncrypt = Base64UrlEncoder.Encode(ssJoseHeader) + &quot;.&quot; + Base64UrlEncoder.Encode(ssJwtClaims);<br>            byte[] bytesToEncrypt = new UTF8Encoding().GetBytes(textToEncrypt);</pre><pre>var blockSize = encryptEngine.GetInputBlockSize();</pre><pre>List&lt;byte&gt; output = new List&lt;byte&gt;();</pre><pre>for (int i = 0; i &lt; bytesToEncrypt.Length; i += blockSize)<br>            {<br>                var size = <br>                    bytesToEncrypt.Length &lt; blockSize ? bytesToEncrypt.Length :<br>                    i + blockSize &gt; bytesToEncrypt.Length ? bytesToEncrypt.Length - i :<br>                    blockSize;</pre><pre>output.AddRange(encryptEngine.ProcessBlock(bytesToEncrypt, i, size));<br>            }<br>            <br>            ssJwt = textToEncrypt + &quot;.&quot; + Base64UrlEncoder.Encode(output.ToArray());<br>  } // MssJwtCreate</pre><h4>Verifying a JWT</h4><pre>public void MssJwtVerify(string ssJwt, object ssCertificate, out string ssJoseHeader, out string ssJwtClaims, out bool ssIsVerified)<br>        {<br>            string[] parts = ssJwt.Split(&#39;.&#39;);<br>            if (parts.Length != 3)<br>            {<br>                throw new Exception(&quot;Invalid JWT: doesn&#39;t contain three parts&quot;);<br>            }</pre><pre>ssJoseHeader = Base64UrlEncoder.Decode(parts[0]);<br>            ssJwtClaims = Base64UrlEncoder.Decode(parts[1]);</pre><pre>var c = DotNetUtilities.FromX509Certificate((X509Certificate)ssCertificate);</pre><pre>var encryptEngine = new Pkcs1Encoding(new RsaEngine());<br>            encryptEngine.Init(false, (RsaKeyParameters)c.GetPublicKey());</pre><pre>var blockSize = encryptEngine.GetInputBlockSize();</pre><pre>byte[] bytesToDecrypt = Base64UrlEncoder.DecodeBytes(parts[2]);<br>            string output = &quot;&quot;;</pre><pre>UTF8Encoding ue = new UTF8Encoding();</pre><pre>for (int i = 0; i &lt; bytesToDecrypt.Length; i += blockSize)<br>            {<br>                var size =<br>                    bytesToDecrypt.Length &lt; blockSize ? bytesToDecrypt.Length :<br>                    i + blockSize &gt; bytesToDecrypt.Length ? bytesToDecrypt.Length - i :<br>                    blockSize;</pre><pre>output += ue.GetString(encryptEngine.ProcessBlock(bytesToDecrypt, i, size));<br>            }</pre><pre>ssIsVerified = output.Equals(parts[0] + &quot;.&quot; + parts[1]);</pre><pre>} // MssJwtVerify</pre><p>¹There are more algorithms defined, but these two are by far the most common.</p><p>²We have an on-premise installation of the Platform, with access to the Windows servers it runs on. Therefore we store the certificates in the Windows certificate store. Of course, of this is not desirable, e.g. when you have a cloud infrastructure, there are other ways to provide a certificate.</p><p>³A <a href="https://en.wikipedia.org/wiki/Basic_access_authentication">basic authentication</a> string starts with the word “Basic”, followed by a space, followed by a Base64-encoded string of the username and password concatenated with a colon (“:”) as separator. If you allow colons in the username and/or password, you’ll need to encode them somehow, otherwise you cannot safely determine where the separator is. I used a backslash (“\”) to escape the colon.</p><p>⁴Note that currently, we’ve made no provision for expired tokens. Ideally, when a token is expired, the customer portal would automatically request a new token, then retry the call it made. But unfortunately that’s not easily possible using OutSystems, except for implementing such functionality for every method that’s called.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fded258cbe14" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/exposing-an-outsystems-rest-service-with-an-oauth-style-authorization-fded258cbe14">Exposing an OutSystems REST service with an OAuth-style authorization</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Timers and when they run]]></title>
            <link>https://itnext.io/timers-and-when-they-run-5614d683c973?source=rss-f3f3765190c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5614d683c973</guid>
            <dc:creator><![CDATA[Kilian Hekhuis]]></dc:creator>
            <pubDate>Fri, 11 Jan 2019 11:20:02 GMT</pubDate>
            <atom:updated>2023-05-02T11:40:26.588Z</atom:updated>
            <content:encoded><![CDATA[<h3>OutSystems Timers and when they run</h3><p><em>This is an article about the OutSystems Platform, and specifically, about Timers (and when they run).</em></p><p>Update: I have added a paragraph that explicitly describes what happens when you manually (or programmatically) start a Timer, and that Timer is currently deactivated. Thanks to Rui Barradas for signaling this omission.</p><p>Update 2023–05–02: Fixed a broken link to a course on timers.</p><h3>What is this Timers you speak of?</h3><p>Timers are the OutSystems Platform’s batch jobs¹. They run asynchronously, are allowed to run for a long time, and are typically used to churn away at data. They are invaluable to any serious business using the OutSystems Platform. Lately, I’ve seen much confusion about how Timers are handled by the Platform, when they are scheduled, and how often they are restarted in case of failure, so I decided to write an article that seeks to clarify all that.</p><p>One of the central components of the OutSystems Platform is the <em>Scheduler</em>. The Scheduler is, ultimately, a Windows Service called <em>OutSystems Scheduler Service</em>. In Service Center, the Environment Configuration page of the Administration menu shows its status: there should be a happy green check mark below it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*vYQx8vGMfsASwtNLCO63dg.png" /><figcaption>There should be a happy green check mark below “<em>Scheduler”</em>. In fact, there should be a lot of happy green check marks on the “Environment Health” page.</figcaption></figure><p>In order to describe what the Scheduler does though, we first need to take a look at Timers themselves, and how they can be configured.</p><h3>Timers, timers everywhere</h3><p>Timers are created in Service Studio like any other logic. To create a Timer, select the Processes Tab, right-click the “Timers” folder and select “Add Timer”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*m1zg4zmY3C3V37MJV0Vh0g.png" /><figcaption>This is how you do it baby…</figcaption></figure><p>The first thing to notice after creating a new Timer is that it has several properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*vMndvRTi0dYjDZlCS73Q_w.png" /><figcaption>Timers have properties, like most objects in the Tree.</figcaption></figure><p>Worth taking a closer look at are the following:</p><ul><li>Name: the Timer’s name. It is used as part of the Timer’s Wake Action;</li><li>Action: this is the Server Action that is called when the Timer is started. It contains the logic of the Timer. If you select <em>New Server Action</em>, Service Studio creates a new Server Action, with a name equal to the Timer’s name. If you subsequently change the Action’s name, the Timer’s name is also updated. If you then change the Timer’s name, luckily the Action’s name stays put;</li><li>Schedule: this is the Timer’s schedule, which defines how often the timer will run and when. Note that any time specified here is the <em>local time of the server</em>, so it’s a good thing to know in which time zone it runs;</li><li>Timeout in Minutes: this is the maximum time allotted to the Timer. If it runs longer, it will be killed off;</li><li>Priority: the timer’s priority. When there are more Timers ready to run than the maximum amount of Timers that are allowed to run consecutively (of which more later on), the Timers with the highest priority get to run first.</li></ul><p>The initial schedule is set via the schedule editor pop-up. This allows for easy creating of a schedule. Note, however, that the schedule can be overridden later on in Service Center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*0yX8fyNf08Ian7zrJAxo_Q.png" /><figcaption>The schedule editor pop-up. If you say “shedule” instead of “skedule”, you are likely anxious about the imminent Brexit. If not, you should be.</figcaption></figure><p>The “When Published” option allows the Timer to run after its Module has been published, which is ideal for stuff like bootstrapping.</p><p>There are a number of <a href="https://success.outsystems.com/Documentation/Best_Practices/Performance_Best_Practices/Performance_Best_Practices_-_Logic#Avoid_long-running_timers_and_batch_jobs">best practices</a> for how to <a href="https://www.outsystems.com/training/lesson/1133/master-class-on-best-practices-and-timers">properly design</a> the logic of a Timer, with regards to committing data, checking for imminent demise, etc. but those are out of scope for this article.</p><h3>…and action!</h3><p>Once the Timer logic has been written, tested and deployed, it’s ready for the Timer to run. If a schedule was specified during development time, a newly created Timer will get that schedule. If no schedule was specified, the Timer won’t run at all, until you manually specify a schedule in Service Center. To do so, navigate to the Module’s page (via Factory / eSpaces), click the Timer tab, and click the specific Timer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*IAZPVsmY1qPlkdwOtuVNWg.png" /><figcaption>For historical reasons, Modules created in Service Studio are called “eSpaces” in Service Center. Paulo thinks this is the worst name choice ever.</figcaption></figure><p>On the Timer’s details page, you are greeted with a myriad of options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*5ZbPx-OYYXQm06aeTo-Xbg.png" /><figcaption>Perhaps not a myriad, but still a lot. If anything, this shows that UI design should not be thought of lightly.</figcaption></figure><p>At the top, there’s the Timer’s <em>Status</em>: it shows whether the Timer is <em>activated</em>, i.e. ready to run, or <em>deactivated</em>, i.e. it won’t run, no matter what the schedule says. Next, there’s the <em>Default Schedule</em>. This is the schedule that was set at development time as explained above. The <em>Effective Schedule</em> is, as the name implies the schedule that’s currently active. It shows the same parameters also shown in the Service Studio pop-up (but with slightly different names to add to confusion, e.g. the pop-up’s <em>Occurs</em> is the Servive Center’s <em>Days</em>), and like the pop-up, the available information changes with the <em>Days</em> setting.</p><p>A neat option that Service Center offers but Service Studio does not, is to generate time intervals at which the Timer runs, e.g. every ten minutes. The minimum interval is five minutes, which makes sense, as Timers are meant for batch jobs that take some time.</p><p>The page also shows when the Timer ran last, and how long it took, and also (if it isn’t deactivated) when the next run is scheduled. If it is actually running right now, that’s also shown (<em>Running By</em> is the date/time when it started, not the person who started it).</p><p>Below all the information there are five buttons. These are:</p><ul><li>Apply: save any changes you made to the schedule;</li><li>Cancel: cancel any changes you made;</li><li>Refresh: refresh the page. Useful for refreshing the “Last Run”, “Next Run” and “Running Since” information;</li><li>Run Now: directly start this Timer, regardless of schedule or status;</li><li>Activate/Deactive: as the name implies, activates a deactivated Timer, or deactivates an active Timer.</li></ul><p>Apart from via the schedule and the “Run Now” button on the Timer’s details page, it’s also possible to run a Timer programmatically. This is done via the Timer’s Wake Action, which is shown below it in the Tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*KS2PHsQK4RGyPeh1m3Lukw.png" /><figcaption>Wake up, you sleepy head, put on some clothes, shake up your bed…</figcaption></figure><p>You can drag the Action onto the canvas, or select it via the Select Action pop-up, where it is available below the Timers folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*vgVIiIQYxWS_ylG4xho5aA.png" /><figcaption>The Timer Timer_Timer1 is below the Timer folder. Now say that quickly ten times.</figcaption></figure><p>This comes in handy when you want a user to be able to start a Timer, e.g. when the user uploaded a large Excel file that needs to be processed, or inside a Timer Action itself, when on pending demise, it restarts itself and then Ends. The way these “manually” started Timer’s interact with the scheduled ones is described later on.</p><h3>Configuring like a boss</h3><p>Now that we know there’s a Scheduler, and how to create Timers and schedule them, let’s take a look at how the Scheduler schedules Timers.</p><p>First, there’s a few configuration options that need to be taken into account, the first of which is the number of Timers that can run concurrently. The Platform imposes a limit on the number of different Timers that can run at the same time, for performance reasons. You wouldn’t want twenty badly scheduled Timers to run all at once, causing your server to come to a grinding halt. By default, three Timers may run in parallel. Note that these are three <em>different</em> Timers. The same Timer can <em>never</em> run more than once at the same time. If you need concurrent data processing, BPT is the way to go. If you want to configure a different number, you need access to the Platform Configuration Tool. On the Scheduler tab, there’s a <em>Max. Concurrent Timers</em> setting².</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*5jPlwv2c14R263EQpVUNgA.png" /><figcaption>So much room, so few options…</figcaption></figure><p>Secondly, there’s the number of execution attempts. This number specifies how often the Timer is run in case it fails while running. “Failing” here means either an uncaught Exception occurs, aborting the Timer prematurely, or the timer is terminated by the Scheduler because it didn’t finish before the configured time-out. In either case, if the number of execution attempts is set to a value larger than 1 (default is 3), the Scheduler schedules the Timer again, until it either succeeds or the number of attempts is up.</p><p>The number of execution attempts is configured in Service Center, on the <em>Environment Configuration</em> page (<em>Administration</em> menu), <em>Timer Execution Attempts</em> setting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*YVucIkGeFHhM5onGyxJNHg.png" /><figcaption>If the Timer is still alive after three attempts, they may go free.</figcaption></figure><h3>Schedulin’ like there’s no tomorrow</h3><p>Like discussed above, every Timer that’s enabled and has a schedule is up for scheduling by the Scheduler. As soon as a a Timer becomes Active (or is published for the first time), the Next Run date/time is calculated and stored. The Scheduler frequently looks (several times a minute) at all Active Timers’ schedules, and determines the ones that are elligible for running by comparing the Next Run date/time with the current date/time. Any Timer whose Next Run is equal to, or older than, the current date/time is put in the Scheduler’s internal run queue.</p><p>Next, the Scheduler looks at the number of Timers in the queue, sorted by Priority (lowest first) and Next Run date/time (earlier first). If there are less Timers currently running than the maximum number of concurrent Timers configured, the first eligible Timer is scheduled for running immediately, and the Scheduler will instruct the Platform to run it³. After the Timer has been started, the next elligible Timer is considered (and started if possible), and so on.</p><p>Note that, keeping in mind the above rules, the actual time a Timer runs is only a “nice to have” time — there’s no guarantee the Timer will actually run at the Next Run date/time, as the Scheduler schedules Timers best effort; the actual date/time might deviate significantly (especially if there are more than the configured consecutive number of Timers eligible).</p><p>When a Timer successfully finishes execution (i.e. on time and without throwing an uncaught Exception), the Scheduler calculates the Timer’s next run date/time (if the Timer has a schedule that’s not “When Published”) and updates its Next Run accordingly (after which it is again partakes in the elligibility checks as outlined above), <em>unless</em> there’s already a Next Run set that’s not the one used for starting the current Timer. The latter is important in case of manual Time runs (see below).</p><p>When a Timer does not finish successfully, the Scheduler checks how many attempts have been already made to run the Timer. If the Timer has reached its maximum number of execution attempts, the Scheduler gives up, and — like for succesful executions — calculates and updates the Next Run.</p><h3>Mixing it up</h3><p>The above is all rather straightforward, but there’s one situation that’s slightly more complex: the earlier described “manual” Timer runs. The way these are handled is pretty trivial though: when a WakeTimer Action is executed, the Next Run is overridden with the current date/time, making the Timer immediately elligible for running. After the Timer is finished, the Scheduler calculates the Next Run just like after any other Timer run (unless there was another manual run scheduled, as explained above). This results in the following behaviour:</p><ul><li>When a Timer is running because of its schedule, and a manual run is requested, the Next Run is set to the current date/time. When the Timer has finished, the Scheduler detects that the Next Run has been altered since the Timer was started, and therefore doesn’t recalculate the Next Run. This makes the Timer directly elligible for running (so it will start again soon after finishing). When the “manual” run has finished, the Next Run will be set according to the Timer’s schedule (unless, during <em>its</em> run, another manual run is requested).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*EgknqkagYt7g1J6ndPCv_g.png" /><figcaption>A single manual start while the Timer is running postpones the manual run until after the current run has finished. After the manual run has finished, a new run is scheduled according to the schedule.</figcaption></figure><ul><li>When a Timer is running, starting it manually multiple times just updates the Next Run multiple times. As soon as the Timer has finished running, it becomes elligible again, and will be run one more time (unless, again, during <em>its</em> run, another manual run is requested).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*m1Pk_30WLG7rfOsWkT8C6A.png" /><figcaption>Multiple manual starts while the Timer is running only cause a single manual start. There is no difference with starting it manually only once.</figcaption></figure><ul><li>When a Timer is scheduled, but before the date/time of the Next Run a manual run is scheduled, the Next Run will be overwritten. When the Timer finishes, if the “old” Next Run is still in the future, the Scheduler will again calculate and store that date/time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*snLnwCikVjPivpJgPbndUg.png" /><figcaption>After the manual run finishes, the Next Run is calculated according to the schedule.</figcaption></figure><ul><li>However, if the old Next Run is in the past (which means the manual run was still running when normally the scheduled run should have taken place), that run is skipped, as the Scheduler will calculate the Next Run based on the schedule, and therefore always in the future.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*zRVjHKpcGENSU6H7CtLQ-w.png" /><figcaption>If a Timer should have run according to the schedule, but it is still running because of a manual start (or, for that matter, a scheduled start), the scheduled start is skipped (actually, its Next Run is overwritten). Once the manual run has finished, the Next Run is calculated according to the schedule.</figcaption></figure><p>There’s one situation that you should be aware of that might catch you off-guard: when a Timer is deactivated, and you run it manually, the scheduler will calculate a new Next Run, as explained above. If the timer has a schedule, this means it will be activated again according to that schedule! The reason for this is that a deactivated Timer actually has a Next Run of 3000–01–01, which is overwritten when the Timer is started manually by the current date/time, as explained above. So after that, there’s no way for the scheduler to know the Timer was deactivated, and so if the Timer has a schedule, the scheduler will calculate the Next Run according to that, effectively activating the Timer! Only when there’s no schedule, the scheduler will set a Next Run of 3000–01–01, and the Timer appears Deactivated again after the manual run. So if you have a Timer that really only must to be started manually, make sure it doesn’t have a schedule.</p><p><em>Addition 2023–02–17:</em> One specific case that I didn’t describe above, and that was pointed out to me today by Eduardo Jauch, is when the default schedule is set to “When published”, but the effective schedule is set to something else. In this case, and this case only, the default schedule is not ignored, and when the module is published, the timer is run. Since a “when published” run is treated as a manual run, after the “when published” run, the scheduler calculates a new Next Run, as described above for a manual run.</p><h3>Run Timer, run!</h3><p>Now that we’ve established how and when Timers run, you may ask yourself “but <em>where</em> do they run?” This seems like a trivial question: if you have a single front-end server, that’s where they run. However, in case you have multiple front-end servers, it becomes less trivial: it is possible to designate a single or multiple servers for running Timers, while others won’t run Timers. To configure this, Service Center has the <em>Front-end Servers</em> configuration page (<em>Administration</em> menu). When you access the page, it has an overview of all configured front-end servers, showing whether they are enabled, whether they execute Timers and Processes, and whether they send e-mails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*Qv2g0rpon4Zuj32aTunDKQ.png" /><figcaption>Front-end Server overview. srv37 is not allowed to Execute Timers. Bummer!</figcaption></figure><p>When you click on a server name, Service Center shows the details page, allowing you to configure the various options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*rNibDVF7U_3pfaTdsp5LUA.png" /><figcaption>See that unchecked checkbox? I know you wanna check it, running Timers rules!</figcaption></figure><p>Note the “Execute Timers” checkbox: as the help text below it says, if it’s checked, the server can run timers, if it’s not, it can’t. That is to say, the Scheduler service still runs on the front-end, but it knows it’s not allowed to schedule Timers.</p><p>If you have multiple front-ends that can run Timers, you never know where they end up running. The Schedulers of these front-ends collaborate to run timers, so that not a single server will end up running all Timers. The precise algorithm is a black box, and probably only known to the OutSystems engineers maintaining the Scheduler service (though it might be as simple as first-come-first-serve).</p><p>So why would you want to deny a server to run Timers? Well, most obviously for performance reasons. A front-end server that’s busy serving pages to users may not be the best server to run heavy batch jobs on. So it is common practice to have one or more servers dedicated to serving user pages, and another one or more dedicated to running Timers.</p><h3>I spy with my little eye…</h3><p>In an ideal world, everything is running a-ok, and there’s little need to keep an eye on Timers: if they are programmed well, and performance of the infrastructure is fine, they just churn away and do what they’re destined to do. However, in such an ideal world, there are probably also rainbow-coloured unicorns frolicking about — and unfortunately, I’ve never come across them. In reality, it is pretty important to be able to keep an eye on Timers and whether they’re running fine.</p><p>There’s several places in Service Center where you can see what’s going on. First and foremost, there is of course the <em>Error Log</em>. If a Timer runs into trouble, the Error Log shows the offending error, and the Scheduler Service will tell whether a retry will be scheduled or not⁴.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*i-cFrH94PCjrzb1GiaL3lg.png" /><figcaption>Actuall, the log is a lie: there’s only three runs total, so two retries, not three.</figcaption></figure><p>Clicking on Detail may tell you a little more about what caused the Timer to abort.</p><p>Secondly, there’s the already mentioned <em>Environment Health</em> page. Apart from showing whether the Scheduler is up and running, it also shows a list of twenty Timers that are next in line for running, and any Timers that are actually running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L8E3Hxo-nnnAmQJPzFTLvA.png" /><figcaption>As opposed to a box of chocolates, the Timer list shows exactly what you’re gonna get.</figcaption></figure><p>Thirdly, there’s the <em>Timer Log</em>. The Timer log shows all Timers that have finished running (whether succesful or not), with their starting time, original scheduled time (“Should Have Run At”) and their Next Run. If they were not succesful, an “error” link is present, which redirects to the same page as the one from the Error Log.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uIpGkxHkiChhC6i9ale1tQ.png" /><figcaption>Everything’s well, no error in sight.</figcaption></figure><p>So, that about wraps it up. If you have any questions left about Timers, the <a href="https://www.outsystems.com/forums">OutSystems Community Forum</a> is the place to head to. If really needed, you can <a href="https://www.outsystems.com/profile/zbx7fxyfxj/overview">PM</a> me there as well.</p><p>¹ Or, if you prefer, cronjobs or tasks.</p><p>² Since this tool runs on the server itself, you cannot access it if your Platform is cloud-based. In that case, contact OutSystems Support if you need to change it.</p><p>³ Since the Scheduler is a Windows Service, it cannot start Timers directly. Instead the Platform exposes a dedicated timer web service, which is called by the Scheduler when it needs to start a Timer.</p><p>⁴ Note that the way the Scheduler schedules retries is a bit odd: since the property that specifies the maximum tries is called “Timer Execution Attempts” (as shown earlier), and the first time the Timer is started is also the first attempt, the number of retries is actually one less than the number of attempts (the first retry is, after all, the second attempt). Nevertheless, the logging shows that the Scheduler actually schedules that number of <em>retries</em>, but at the start of the last retry directly aborts, since the maximum number of retries has been reached (as is also visible in the logging, see the screen shot), after which the next run is calculated (that is the next “normal” run, not a retry).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5614d683c973" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/timers-and-when-they-run-5614d683c973">Timers and when they run</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consuming a multipart/form-data REST Method with OutSystems]]></title>
            <link>https://itnext.io/consuming-a-multipart-form-data-rest-method-with-outsystems-c466e404118d?source=rss-f3f3765190c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c466e404118d</guid>
            <category><![CDATA[outsystems]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[rest-api]]></category>
            <dc:creator><![CDATA[Kilian Hekhuis]]></dc:creator>
            <pubDate>Wed, 11 Jul 2018 06:17:45 GMT</pubDate>
            <atom:updated>2023-02-22T17:23:36.423Z</atom:updated>
            <content:encoded><![CDATA[<p>Update: there were some technical inaccuracies in the article, which I have fixed.</p><p>Update (2023–02–22): OutSystems has announced that in Service Studio 11.53.40, together with Platform Server 11.19.0, the functionality described below is now a native part of OutSystems REST consume. I’ll let this article stand for now, I might update it later with the native functionality.</p><h4>Preface</h4><p>A couple of weeks ago, I was asked to create in OutSystems an interface with a REST API of an external supplier. Except for a terrible lack of functional documentation on the side of the REST API’s supplier, the process was pretty straightforward until I hit a stumbling block: one of the methods, for uploading documents, used something called “multipart/form-data”, which is not natively supported by the Platform. Being pretty active on the <a href="https://www.outsystems.com/forums">OutSystems Community Forum</a>, I have regularly seen people running into this problem when trying to consume a REST service (see e.g. <a href="http://xn--https-ix3b//www.outsystems.com/forums/discussion/17055/rest-api-multipart-form-data">here</a>, <a href="http://xn--https-ix3b//www.outsystems.com/forums/discussion/32907/multipart-file-upload">here </a>and <a href="http://xn--https-ix3b//www.outsystems.com/forums/discussion/33380/how-to-consume-http-post-rest-api-using-multipart-form-data-method">here</a>), but had never experienced it myself, until now. So I did two things: first, I created a <a href="https://www.outsystems.com/ideas/5022/rest-native-support-of-multipart-form-data">Community Idea</a> for OutSystems to support it natively, and secondly, I came up with an idea of how to satisfy the REST API with minimal impact for my implementation of its interface, which I’ll describe in this article.</p><h4>REST in OutSystems</h4><p>The first thing I want to look at is the way OutSystems implements REST. REST is a protocol on top of HTTP (the protocol used for retrieving web pages), and typically (though not exclusively) uses JSON to communicate data. When sending data via HTTP, the sender must specify the type of content it is sending. Since OutSystems support JSON only¹, it sends “application/json” in the HTTP “Content-Type” header. In the body of the message, the actual JSON is sent, in a compact manner (no line breaks), like this:</p><pre>{&quot;name&quot;:&quot;Test for POST&quot;,&quot;type&quot;:&quot;PACKAGE&quot;,&quot;language&quot;:&quot;en&quot;,&quot;description&quot;:&quot;This is a test for posting a package&quot;}</pre><p>This works for probably 95% of REST methods, so it’s in line with OutSystems philosophy of supporting the most commonly used software patterns, without the want to support them all.</p><h4>Multiwhat?</h4><p>So what is this “multipart/form-data” I talked about earlier, and why is it used? First some background. “Form data” is HTTP terminology for any data a user enters on a web page (“HTML form”) and which is subsequently sent (or “posted”) to a web server via HTTP. Typically a form contains only simple input fields, and those are sent in a format that you might recognize, as it is the same format as used for passing parameters to web screens: it consists of name/value pairs, separated by an assign character (“=”), concatenated by an ampersand (“&amp;”) with special characters replaced by a percentage sign and their hexadecimal character code (e.g. “%20” for space):</p><pre>name=This%20is%20a%20test&amp;language=en</pre><p>The HTTP content type for this format is “application/x-www-form-urlencoded”. This works fine for simple input fields, but web pages may also allow the user to upload one or more files. To support this, another format was created in the early days of the web, and it is known by its content type, “multipart/form-data”. So in essence, multipart/form-data is a way for a browser to send one or more files to a web server (it was later co-opted for sending attachments in e-mail, and even later for REST). The exact format for sending multipart/form-data is rather arcane (as are most early web protocols), but fortunately for us it’s mostly text only, so it’s easy to read and manipulate.</p><p>First, instead of just the content type, a client (like a browser or REST consuming app) sending multipart/form-data sets the HTTP Content-Type to “multipart/form-data”, but also needs to specify something called a <em>boundary</em>. This is a unique text string that must be guaranteed not to appear anywhere in the message, so it can signal the several sections or parts (it’s <em>multi</em>part, after all). It is customary to start the boundary with a string of dashes followed by some kind of GUID, though a client is free to set the boundary to whatever it likes (within limits), as long as the rule I stated above is met. The boundary is added after the content type with a semicolon, followed by the text “boundary=”, followed by the boundary itself, between double quotes:</p><p>Content-Type: multipart/form-data;boundary=&quot;MyBoundary&quot;</p><p>The body of the message consists of one or more section, each part starting with two dashes (--) followed by the boundary text. On the next line, there’s the text Content-Disposition: form-data; name=&quot;<em>name</em>&quot;, where <em>name</em> between quotes is the name of the item. If the section contains file data, the line also contains the filename, like this:</p><pre>--MyBoundary<br>Content-Disposition: form-data; name=&quot;myFile&quot; filename=&quot;myfile.docx&quot;</pre><p>After this line there’s a blank line, followed by the value associated with “<em>name</em>”. For simple input fields this is the value of the input field, for files this is its binary data. Directly after the data, a new section starts (without a blank line in between). After the last section, to signal the end of the multipart/form-data message, the boundary is added again, but this time not only starting, but also ending, with two dashes:</p><pre>--MyBoundary--</pre><p>Of course, I’m not the first one to attempt to solve the problem of sending multipart/form-data. Forge Components for various REST APIs have solved it in a number of different ways, and there even is a Forge Component for <a href="https://www.outsystems.com/forge/component/3731/multipart-upload">sending multipart/form-data</a> independently from REST. But none of what I have seen so far was entirely to my liking. So I decided to create an implementation that suited my needs (and my taste for aesthetics).</p><h4>Implementation details</h4><p>Whatever implementation one choses, in the end one needs to do two things: send the right Content-Type header, and send the correctly formatted body (as explained above). If you include files containing binary data (i.e. anything other than a text file), you need to declare the REST API Method having a Parameter with Data Type “Binary Data”, and its “Send In” Property set to “Body”. This allows sending the binary data unmodified, which is what we want. If also prevents the Platform from adding a Content-Type tag in the Header, which you then need to add (via a Parameter with “Send In” set to “Header”). The (small) downside is that in order to compose the message, you need the <a href="https://www.outsystems.com/forge/component/2067/binary-concat/">Binary Concat</a> Forge Component in order to be able to concatenate the various parts, as well as Actions from the BinaryData Extension (which is a System Component, so already installed) to convert the text-based parts of the message to binary before concatenating it with the binary data of the files.</p><p>Another approach, which only works when the message does <em>not</em> contain any binary data (e.g. when sending text files only), is to declare the REST API Method as a “normal’ JSON one, and modify the message before it is sent to the REST service. In OutSystems, this is done by defining an OnBeforeRequest Action. An OnBeforeRequest Action is defined in the Advanced Properties of a REST API. Chose “New OnBeforeRequest” from the dropdown menu (not the “Advanced” one!), and below the REST API the Action will appear:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*hNQWKTIeZT8ejJyWYEejJA.png" /><figcaption>Select “OnBeforeRequest” for great justice!</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*U8g-4bDe8DUtDgdI-HsxVQ.png" /><figcaption>There it is, for your modifying pleasures.</figcaption></figure><p>As can be seen in the image, the OnBeforeRequest has one Input Parameter, and one Output Parameter. Both are of type HTTPRequest, and contain everything you need to modify the REST message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*sILCCcH1Fl1VQ_0mkRFLcQ.png" /><figcaption>The HTTPRequest Structure.</figcaption></figure><p>Note that the OnBeforeRequest is a generic way of modifying the REST message, and not specific to multipart/form-data. Also note that there is a single OnBeforeRequest for the entire REST API, not one per Method. This means that inside the OnBeforeRequest, you need to check which Method is called by inspecting the URLPath and HTTPMethod Attributes of the Request Input Parameter².</p><h4>My solution</h4><p>As I needed to send binary data, I had to choose the first approach³. This meant I added two Input Parameters: one for the content type, and one for the binary body. Using my Forge Component (see below) I added the multiple parts (including a JSON part) to the message, after which I assigned the concatenated parts via the Binary Data parameter.</p><h4>Forge Component</h4><p>Regardless of the specific solution that makes sense for your project, I created a Forge Component, aptly name <a href="https://www.outsystems.com/forge/4026/">Multipart/form-data</a>, that consists of a single eSpace with a number of Public Actions that aid in creating the right content:</p><ul><li>MultipartFormDataCreate — the main Action, creates a text body consisting of the specified parts, in a multipart/form-data compatible format.</li><li>ContentTypeGet — returns the value to be used for the Content-Type header of the REST message based on the boundary used.</li><li>PartAdd — a helper Function that can be used to easily create a List of parts.</li></ul><h4>Wrapping up</h4><p>That was it, my first article on Medium. I hope you like it, and find it useful when implementing REST with multipart/form-data. Any questions can be asked on the <a href="https://www.outsystems.com/forums">OutSystems Community Forum</a>, the <a href="https://www.outsystems.com/forge/component-discussions/4026/Multipart%2fform-data">component’s subforum</a>, or as a last resort via <a href="https://www.outsystems.com/profile/5974/">PM</a>.</p><p>[1] This is not entirely true for consuming REST services, but for the sake of this article we’ll assume it is, as the other supported types are of no use for the “multipart/form-data” we want to support.</p><p>[2] Another solution would be to isolate a single method that needs multipart/form-data in its own REST API, but I would advise against this, as there’s no logical grouping of Methods anymore, and you need to configure the same base URL twice in Service Center, which is prone to errors.</p><p>[3] In a previous version of this article, I mentioned choosing the second approach. This was based on the erroneous assumption I could make it work by converting binary data to text, which turned out to be not the case.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c466e404118d" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/consuming-a-multipart-form-data-rest-method-with-outsystems-c466e404118d">Consuming a multipart/form-data REST Method with OutSystems</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>