Mattias Lögdberg

MSI with Service Bus

2022-03-29T00:00:00+00:00

Using MSI with Service Bus is a great addition to the SaaS key that is the standard authentication mechanism.

What is MSI (Managed Service Identity)? Read more in previous post. In a short summary it’s a way of giving your resources (can be a Logic App, a Function App etc.) an identity and assing permissions.

In the case of using MSI with Service Bus it can be used with our internal services accessing the Service Bus queues, topics or topic subscriptions.

The Access Roles that are available on Service Bus are the following:

Azure Service Bus Data Owner: Grants access to both send and recieve
Azure Service Bus Data Reciever: Grants access to read messages from queues or topic subscriptions
Azure Service Bus Data Sender: Grants access to send message on queues or topics.

When assigning roles there are some things to think about:

What are the current service suppose to be able to access? All queues or specific queues?
If you use Functions or Logic Apps as Reciever of messages from either queue or topic subscriptions you must have assigned the Azure Service Bus Data Reciever on the specific queue or topic subscription. It will not work to assign it on the whole namespace.

Let’s take an example of reading messages from a queue with a Logic App. We are creating a consumption Logic App and the first step we need to do is assignt an identity to the Logic App.

Now we need to Assign Roles to the Logic Apps Identity to the Queue, remember that it has to be on the Queue in order to work with recieving messages.

Go to the Service Bus namespace and to the Queue, in my case Messages and add the role assingment Azure Service Bus Reciever to our Logic App.

Pick the role and then select the Managed Identity that is connected to your Logic App

Now we can create a trigger in the Logic Apps Designer as follows:

If you have assigned the Role on the queue you are now ready to start recieving messages.

Removing the access SaaS key

There is an option to remove access with the SaaS key. This is preferable if the Service Bus namespace can be used with resources who can use Managed Idenity or Service Principal when authenticating to the Service Bus Namespace. If the SaaS key access is removed the whole namepsace is only accessible with a valid Aure user. This is a much securer way and moves the access control to Azure AD rather than a Service Bus specific authentication mechanism.

To close the Access Key go to the namespace and press the Local Authentication: Enabled link and change to Disabled. After this only access via Azure AD Roles will be working.

Troubelshooting

If no messages are recieved check the triggerhistory and verify that there are no errors.

Summary

Using Managed Identity or Servcei Principal with Service Bus is a great feature, it’s much easier and secure. No static keys that can wander of and no unkown users. Developers accessing the namespace have to have valid Azure AD accounts that have assigned permissions. I know it can be tricker in the start but in the long run it will help out alot.

MSI what and why?

2022-01-19T00:00:00+00:00

MSI or Managed Service Identity that it stands for has been around for some time but quite recently a new set of resources has been integrated in to this and we need to start thinking of using MSI alot more. Since this will increase security and streamline the authorization process, one place to grant access, same account and authorization process. Sadly most of Microsoft’s documentation around this is still for Virtual Machines and I want to enlighten this alot more for IPaaS services like Logic Apps, Functions, API Management, Web Apps, DataFactory Service Bus and Event Grid.

So MSI is an identity of a resource just like any other user in Azure AD. The beauty of this is that with this we can assign access rights to that identity and the resource now authenticates by itself (with credentials no one else nows) and with those credentials it can use the resource. I usually like to discribe it as a virtualization of the service and used for authorization purposes, in the image bellow we can see that an Azure Function are making requests to Cosmos. This is allowed since we have assigned the role Cosmos DB Built-in Data Contributor to the identity of the function in a scope containing the Cosmos resource (could be on resource itself, a resource group that grants access to all cosmos resources in the resource group, or subscription granting access to all cosmos resources in that subscription).

What Resources can use MSI: (there are more but these are Integration focused)

Logic Apps,
Azure Functions
API Management
Web Apps
DataFactory
Event Grid Read the full list here.

What resources can we use authenticate to with MSI.

Azure SQL
Cosmos
Service Bus
Event Hub
Azure Functions (if you use Azure AD as provider in Authentication)
Web Site (if you use Azure AD as provider in Authentication)
Logic Apps (If you use Oauth Authorization)
- In Logic Apps we can use MSI to connect to resources like Event Grid for creating subscriptions.

Is this not what we accomplished before?

No before we created a SQL account, took keys from Cosmos, Service Bus etc, create a service principal or a service user in AD to accomplish other acesses. All of these could potentially be simplified and changed to only give the resource the appropiate access rights. Note not all resources is supporting this yet, but this is an area where Microsoft has done some great gains last 6 months.

But isn’t this making my development process a pain?

No on the contrary, you as a developer don’t need to gather all these credentials either, development and testing can be used with your own account. Making it easier to get started and another great benefit is that you cannot accidently keep credentials and access after an completed assignement or project. With good documentation it’s also easier to get a new teammate up to speed. Another good thing is that we know that credentials are not wandering away when members leave the team.

What should I think off

As a developer you could usually grant yourself access to diffrent resources. When using MSI this is no longer the case, since the access is now granted by Azure AD and Azure AD access is often prohibited for developers. Authroization now need’s to be given by a user with Owner or User Access Administrator, to overcome this you need to create a plan for your team how this should be done.

A tip here is to create a Azure AD Group and add all developers to it that need specific access, remove them when they don’t need it. An extra step would be creating a PIM group for it.

Using MSI in APIM to Cosmos

2021-11-22T00:00:00+00:00

Quite recently Microsoft finally gave us RBAC roles for data plane access to Cosmos, this means that we now can use MSI to access data in Cosmos.

So what’s it all about? Well the best part is that we now can grant access to resources rather than using keys. This increases security and accessability since we don’t need to manage the key any more and only specific resources can access our data. When we have switched to RBAC as the authentication method we can switch off the possibility to access data with primary/secondary keys,read more here.

So how do we do this? Here is a guide from Microsoft on how to use MSI with an Azure Function. And we will in this post use the same technique but from API Mangement to show the potential.

Prerequisites:

API Management instance
Identity created on your API Management instance
Cosmos DB
Azure CLI

First we need to grant the specific roles to our API Management identity. The role we want to assing is the Cosmos DB Built-in Data Reader. This role has read access and that matches our Least Previliges thinking, since when we only want to expose read operations we should only have read access.

Adding this role is not as straight forward as I would like it, they are not possible to set via the Azure Portal yet. So we need to do some PowerShell scripting. Fortunally all scripts needed can be found at the Microsoft docs on How to setup RBAC for Cosmos DB. I would recomend looking in there since they also provide how to create your own custom role definitions.

Anyway let’s get started and we need this script:

$resourceGroupName = "<myResourceGroup>"
$accountName = "<myCosmosAccount>"
$readOnlyRoleDefinitionId = "<roleDefinitionId>" # as fetched above
$principalId = "<aadPrincipalId>"
New-AzCosmosDBSqlRoleAssignment -AccountName $accountName `
    -ResourceGroupName $resourceGroupName `
    -RoleDefinitionId $readOnlyRoleDefinitionId `
    -Scope "/" `
    -PrincipalId $principalId

Let’s get the parameters, first we get the cosmos values, the resource group and the account name of the Cosmos instance ($resourceGroupName,$accountName) both marked in circles.

Then as we saw at the Built in role definitions the role with name Cosmos DB Built-in Data Reader has Id 00000000-0000-0000-0000-000000000001

Last we need the principal id of our Api Management instance (this could be any identity created for any resource). We go to the Managed Identity in our API Management instance. There we can collect the Object (principal) ID.

So the script ends up like this: (Make sure to run the latest az version)

$resourceGroupName = "microservice"
$accountName = "mlogdberg"
$principalId = "c7b89cc3-1b27-474b-9575-cdcbc51dffcc"
$readOnlyRoleDefinitionId = "00000000-0000-0000-0000-000000000001" # as fetched from definition list
az cosmosdb sql role assignment create --account-name $accountName --resource-group $resourceGroupName --scope "/" --principal-id $principalId --role-definition-id $readOnlyRoleDefinitionId

After this we are ready to start quering, I did a simple request that collects all documents in a specific partion. This can ofcourse be expanded but this is bare minimum what is needed in the API Managemenet Policy.

Some good to know things:

authentication-managed-identity is the step to get the token, make sure to have the patht to your cosmos instance.
x-ms-documentdb-partitionkey is what partion the query is targeting
rewrite-uri this is the patht to your database and collection /dbs/{dbname}/colls/{collectionname}/docs/
The url is set on the api to the path for my cosmos instance (https://mlogdberg.documents.azure.com).

    <inbound>
        <base />
        <set-variable name="requestDateString" value="@(DateTime.UtcNow.ToString("r"))" />
        <authentication-managed-identity resource="https://mlogdberg.documents.azure.com" output-token-variable-name="msi-access-token" ignore-error="false" />
        <set-header name="Authorization" exists-action="override">
            <value>@("type=aad&ver=1.0&sig=" + context.Variables["msi-access-token"])</value>
        </set-header>
        <set-header name="x-ms-date" exists-action="override">
            <value>@(context.Variables.GetValueOrDefault<string>("requestDateString"))</value>
        </set-header>
        <set-header name="x-ms-version" exists-action="override">
            <value>2018-12-31</value>
        </set-header>
        <set-header name="x-ms-documentdb-query-enablecrosspartition" exists-action="override">
            <value>false</value>
        </set-header>
        <set-header name="x-ms-documentdb-partitionkey" exists-action="override">
            <value>["1"]</value>
        </set-header>
        <rewrite-uri template="/dbs/staging/colls/order/docs/" copy-unmatched-params="false" />
    </inbound>

Now we shall be able to send a request,

Troubelshooting tips:

Request blocked by Auth

Before the setup is completed you will get this kind of respose trying to call the cosmos rest api.

HTTP/1.1 401 Unauthorized
content-location: https://mlogdberg.documents.azure.com/dbs/staging/colls/order/docs
content-type: application/json
    {
    "code": "Unauthorized",
    "message": "Request blocked by Auth mlogdberg : No AAD tenant is trusted by this database account. Please create atleast one Role Assignment prior to making an AAD-based request.\r\nActivityId: 31a968bf-7652-4434-abd9-5113ea0655ae, Microsoft.Azure.Documents.Common/2.14.0"
}

Partition key 1 is invalid

The header x-ms-documentdb-partitionkey is important and when badly configured these kind of errors is sent back, the expected format is a json array of strings, if your partion is “1” the header should have value [“1”] (like my example). You can find your partionkey from browsing your collection.

HTTP/1.1 400 Bad Request
content-location: https://mlogdberg.documents.azure.com/dbs/staging/colls/order/docs
content-type: application/json
    {
    "code": "BadRequest",
    "message": "Partition key 1 is invalid.\r\nActivityId: 6bccbf25-657b-4a71-af92-b3e998f0490a, \r\nRequestStartTime: 2021-11-21T18:24:18.9692549Z, RequestEndTime: No response recorded; Current Time: 2021-11-21T18:24:18.9792812Z,  Number of regions attempted:1\r\n{\"systemHistory\":[{\"dateUtc\":\"2021-11-21T18:23:09.7385423Z\",\"cpu\":8.902,\"memory\":58190802944.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.012,\"availableThreads\":32761,\"minThreads\":40,\"maxThreads\":32767}},{\"dateUtc\":\"2021-11-21T18:23:19.7586326Z\",\"cpu\":7.062,\"memory\":56609513472.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0161,\"availableThreads\":32764,\"minThreads\":40,\"maxThreads\":32767}},{\"dateUtc\":\"2021-11-21T18:23:29.7687307Z\",\"cpu\":5.752,\"memory\":56766377984.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0173,\"availableThreads\":32765,\"minThreads\":40,\"maxThreads\":32767}},{\"dateUtc\":\"2021-11-21T18:23:49.7689519Z\",\"cpu\":5.634,\"memory\":57133617152.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0145,\"availableThreads\":32765,\"minThreads\":40,\"maxThreads\":32767}},{\"dateUtc\":\"2021-11-21T18:23:59.7790729Z\",\"cpu\":5.298,\"memory\":57189658624.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0125,\"availableThreads\":32765,\"minThreads\":40,\"maxThreads\":32767}},{\"dateUtc\":\"2021-11-21T18:24:09.7891858Z\",\"cpu\":8.835,\"memory\":56394321920.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0193,\"availableThreads\":32764,\"minThreads\":40,\"maxThreads\":32767}}]}\r\n, Microsoft.Azure.Documents.Common/2.14.0"
}

Summary:

The usage of MSI (Microsoft Service Identity) is a great feature on Cosmos, no need for storage keys. It’s a win win thing, increased simplicity and also increased security. No credentials are stored and due to that they cannot be leaked. Forcing components to use MSI when connecting to the Cosmos could be done disableLocalAuth to true: "disableLocalAuth": true

AAD Personal Token in requests with VS Code

2021-10-21T00:00:00+00:00

Two weeks ago I did a post around how I use Postman to generate AAD Oauth tokens for Service Principals. Soon after released I started to get questions, THANKS ALOT to all of you who reads and comments. One thing lead to another and an interesting discussion arised with one of you, from that dicussion I had two questions ringing in my head. First is it a good idea to give all developers client credentials to access services? It takes time to create and these credentials is possible to share amongst others, even outside organisations. And second can’t we as a developer just use our own credentials and access?

If we could use our own account instead of a service principal we would increae security, starting time and flexibility all at once.

So I started to look around and I found a good option for this, and that is what I’m going to share today.

I found that VS Code has a nice extension that I will use in this case the Rest Client.

So first install the extension in VS code.

After this, let’s create a new Workspace and start testing this out. As a standard I create a folder and add the requests there, it’s easier to reuse and modify for new requests.

I’ve create a folder called D365 and added a file GetQuotes.http. the .http ending tells the Rest Client extension to VS Code it’s a http request and therefore adds a button on the top Send Request that will trigger the request.

Structure and semantics in this file is very easy, tactic is just write the request in raw format and the press send.

Let’s go thru some small exampels, simple GET example get comment 1, write the Method URL and HTTP protocol to use, so for a GET to https://example.com/comments/1 using standard HTTP/1.1 it will be written as:

GET https://example.com/comments/1 HTTP/1.1

A more advanced exampel, POST example, to create a new comment,POST to https://example.com/comments using standard HTTP/1.1 and we add the json body raw and don’t forgett the content-type header:

POST https://example.com/comments HTTP/1.1
content-type: application/json

{
    "name": "sample",
    "time": "Wed, 21 Oct 2015 18:27:50 GMT"
}

As you can see we just write what we want to send, but enough of the basics let’s get going.

When we want to use our own credentials in order to do calls to an AAD proteced resource like Dynamics D365. We can then use the built in variable $aadToken in the Rest Client, read more here.

That is simple to use we just need to add it to our Authorization header. Let’s look at an example to get Sales Quotation Headers from D365.

GET https://.cloudax.dynamics.com/data/SalesQuotationHeaders?cross-company=true&$top=3 HTTP/1.1
Authorization: 

When we then press the Send Request as shown in the image:

A login prompt will poput like the one bellow, press signin.

A browser will open and you need to paste the code from the previous promt in the code box (it’s in your clipboard at this point so just paste it)

Follow the login procedure and get back to VS Code to see the request been sent and you can see the data.

This request is then sent to D365 with a token generated for you. As with Postman or other tools you can now start building your own collections of requests in VS Code to share, check in with code or just for your own use. Read more in the Rest Client page to get details on advanced request building.

Summary:

As a developer we often need to query systems like D365. When using the “normal” way with service principals we are using credentials anyone can use. We can share them amongst collegues and they will be stored on developers computers. But with the described approach we can stop that and start pushing developers to use their own credentials. Increasing security and control over who has access to our systems and data.
So not only is it awesome for a developer to get started more quicker and easier without the need of a service principal. It’s also a better practice security wise.

There are probobly many more ways to solve this, but this is one way and my thoughts on it.

Oauth Token generation with Postman

2021-10-05T00:00:00+00:00

Beginning to develop solutions where ouath is used as authentication mechanism is often easy due to alot of frameworks sorting out the heavy lifting. But I prefer to go in and see the raw data and make small changes/new requests to the apis to get data in order to understand what’s going on. In those cases I use Postman and build up collections to easily get the data I want and need. And the last few days I’ve gotten questions from multiple people around how I do it so I felt I should share a blog post about it since more might be interested.

First of all this is the way I’m doing it so if you have a better please fill in the chat since there are more than one way to solve this problem. I create a new Workspace “Token generation demo” in Postman (Workspace is a container for i.e. a project where I need to gather all requests and settings I need when working in this project).

Let’s go thru the basics needed, if you never have been working with Postman these are the areas we will use. 1) Collections, here is all our collections that contains requests that we want to group togheter, like “Sales Orders” or “D365” 2) Request area, here we will work with our requests 3) Environmnet variables, these will be used to safely store credentials and also quickly change between environments

There are some built in mechanism but I’m always finding myself in some area of trouble and spend alot of time troubelshooting and just come out confused, so the approach I’m using feels easier and you get more control over what’s going on, call me old school if you want :).

So let’s start of by creating a collection, I name it Token Sample after that we are ready to add our first request, it’s the one for creating the token, we are doing this towards Azure AD. 1) Press the new button and pick ‘Http Request’ 2) Add the url and to Azure it’s always https://login.microsoftonline.com//oauth2/token only switch the to your tenant id. 3) Add the body, since this is where we will send in credentials and all other information needed.

grant_type:client_credentials
client_id:
client_secret:
resource:

What type to use, since we use client credentials we pick client_credentials and then the id and secret of your Azure AD app. And last the audience we are targeting, in this sample it’s the Azure Management api but ut could be any resource that you have in your AD.

4) Name the request so you can find it easily 5) Save it in your collection

Now we have added a bunch of variable, each place that looks like this is avariable with the name that’s inside the curly brackets. Let’s add an environment to add these variables too, clikc on the eye and then the Add button for environment. Start populating the variables we need:

tenantId - your tenant ID can be found in the tageted AD
client_id - the client id of the application you are authenticating with
client_secret - the client secret of the application you are authenticating with
audience - the targeted audience, meaning where do I want to have access, in my case to the Managemet Api.

If we now go back to the request we can generate a token, make sure to pick the environment at top right (see image):

Now we need that token when we do request’s there are several ways to do this but my favourit one is to add a variable for it, and get it updated automatically. Under the tab Tests we can add scripts, read more on scripts. We will then add a simple script that will verify that we get a json body back, and pick out the bearer token in the response and update the environment variable named bearerToken.

var json = JSON.parse(responseBody);
tests["Get Azure AD Token"] = !json.error && responseBody !== '' && responseBody !== '{}' && json.access_token !== '';
postman.setEnvironmentVariable("bearerToken", json.access_token);

And after the run we now have the token in the variable bearerToken as we can see under the environment by clicking on the eye.

Now we can start using it and it’s very simple to start using it, let’s start of by scanning what subscriptions I have access to with this service principal from the Azure management Api.

1) Create a new request 2) Name it Get Subscriptions 3) Set operation to GET and url to https://management.azure.com/subscriptions?api-version=2014-04-01-preview 4) Add the Authorization header with the Bearer type and the token from the variable. 5) Send the Request

As we can see there is a list of Subscriptions returned, and we can now easily setup new environments and reuse the same requests just change the enivornment and the results will be based on the new credentials. This will quickly get you up and running with wathere fun things you are doing.

Summary:

We often need the flexibility that running our own requests gives us, this is how you can use Postman. I often end up building request collections in Postman and distribute them over the team or use them for testing and retesting things. As I usually do I prepared a collection for you if you want to speed up the implementation, download it here

Troubelshooting Azure API Management

2021-01-08T00:00:00+00:00

We often encounter situations where we need to troubelshoot our API’s and in this session I will go thru the different options we have. Following is the 3 different places where we can troubelshoot, here is a quick overview of the three options and here is a link to a video using all these options.

Built in Analytics

In the built in Analytics we can easily see a good overview and status on our API’s. We can see hwo request’s are going, succes, failed? and so on we can also see data transfered and get an insight into response times. Filtering can be done on API’s, Products and even subscriptions to get an insight into how a specific consumer experiences our API’s. Using the Request tab we can even se the requests made, very usefull i you want to find isues like 404 responses or similar.

Application Insights

In addition we can add other tools to get deeper insight into what is happening, i.e why do the consumer get a 401? Or find the error text with that 500 response that is returned. If our backend resource also uses Applciation Insight’s you get a the full end-to-end experience where you can track all relevant information connected to the request like dependencies, logged events and so forth. In Application Insights we can see all failures under the Failures tab.

Clicking you thru we can see the timeline and also the specific error text, in my case there is a TokenClaimValueMismatch at the policy section validate-jwt and if you watch the video we can see that this is exactly why we get the 401. The token provided is missing a permission.

Live Debugging in VSCode

If we need to go deeper in to your troubelshooting in your API Management Policy we can use the Azure API Management VSCode extension. In this extension we can edit our policies for our API Management instance and upload them back when done.

We can start a debugg session where we can send a request and debug the policy step by step! This is awesome and so usefull! (only that request will be captured, so this can safely be done even in production). Right click the Operation and select “Start Policy Debugging”.

The Request is creted and you can add you settings, payload etc and then press “Send Request”.

When using versioning please verify the URL, there is currently no support for the versioning so in my case the versioning is lost my URL is in the sample bellow https://.azure-api.net/Orders/orders/my but due to versioning should be https://.azure-api.net/Orders/v1/orders/my.

For more details and talk, see the video:

Summary:

Beeing a great API publisher is about building robust API’s and taking care of our API Consumers and help them to give a great experience on all fronts, specially when there is issues. We want to be able to find em, fix em and report back to the consumer with either help on what to do to fix it on the consumer side or a “we fixed it” from the API publisher side.

API Management and Auth0 Consumer Friendly API's

2020-06-17T00:00:00+00:00

Today we will extend our setup with API Management even further. Focusing on consumer friendly API’s and in this setup that is knowing our user more. In this example we will add the customer id to the consumer representation in Auth0 and use this id to give the consumer an operation called My orders. No need for knowing or using or adding an id in the request. The id will be added to the token from Auth0 and then extracted in API Management and sent downstreams to filter the result set.

An Identity provider is a service that a user/application is signing in to (just like Azure AD) and this provider has functionality to provide needed information and grant access to requested resources that the IDP is handling. Just like the fact that you have access to a resource or resource group inside your subscription in Azure.

In API Management we have a trust setup as of the previus post Setup Auth0 with API Management. Then we added RBAC permission in the second post RBAC with API Management where we added functionality to provide granular control over who has access to what operation. Now we will continue this and focus on extending this setup with adding a customerid to our consumer representation i Auth0 and then extract that id in API Management to enforce that they only can access these customers orders. Providing an operation to get just my orders without any additional parameters. This is achieved via the validate-jwt policy and then reading the JWT token custom claim and store it in a variable for later use.

<set-variable name="customerid" value="@(((Jwt)context.Variables["token"]).Claims.GetValueOrDefault("https://mlogdberg.com/customerid","-1"))" />

The end scenario will look like the illustration bellow where we can extract the customerid from the token and use it later down the path.

I’ve created a video that will go thru all of this a link is provided bellow.

Links used in the video:

Validate JWT Token

Summary:

Beeing a great API publisher is not as easy at it first looks like. Always striding for better and easier to use API’s is a good thing and here we show how to move more logic inside the setup rather than at the consumer. If we can prevent the consumer from needing to filter data and/or manage identifiers and so forth we can make it easier for the app builders that consume our api’s and provide speed and agility to those teams. At the same time we can create more secure and robust API’s with more automated testing. There are more ways to use this and interested to see what you will find most usefull.

RBAC with API Management

2020-06-12T00:00:00+00:00

Extending our setup with API Management and Auth0 to create RBAC control over our operations in our API’s. The idea is to make sure that consumers can be granted access to specific operations in an API and not only to the whole API. This can be achieved in a number of ways and in this topic we will use Auth0 as the Identity Provider to help out with this, but you could use any other of your choice.

An Identity provider is a service that a user/application is signing in to (just like Azure AD) and this provider has functionality to provide needed information and grant access to requested resources that the IDP is handling. Just like the fact that you have access to a resource or resource group inside your subscription in Azure.

In API Management we have a trust setup as of the previus post Setup Auth0 with API Management. Then we add permissions to our representation of the Api Management instance in Auth0 and grant the conusmer only the permissions that we want to give. After the permissions is setup and granted we willenforce these permissions in our API Management instance via the validate-jwt policy.

The end scenario will look like the illustration bellow where we can grant access to some or all operations in our API.

I’ve created a video that will go thru all of this a link is provided bellow.

Links used in the video:

Validate JWT Token

Summary:

There is alot of times where extra granularity is needed in the API’s exposed. It helps out with creating easier to use and more natural API’s. I can be a good selpoint if used to show that there are more features for premium customers. Regardless adding RBAC to your API’s will increase seurity and eas of use. It will also give you a possibility to move the approving of new customer out of the API Managament instance and become more of a business thing to increas new consumer adoption. The setup used in Auth0 is very powerfull and can be reflected on end users aswell and that can be very helpfull when adding more lightweight consumers with single purposes.

Setup Auth0 with API Management

2020-06-05T00:00:00+00:00

API Management is an awesome API gateway with functionality to really excell in exposing API’s to consumers. When it comes to security there are several options and today we will look in to the OAuth. In order to do this we need an IDP (Identity Provider) that we can configure a trust releationship with.

An Identity provider is a service that a user/application is signing in to (just like Azure AD) and this provider has functionality to provide needed information and grant access to requested resources that the IDP is handling. Just like the fact that you have access to a resource or resource group inside your subscription in Azure.

In API Management a trust to an IDP and creation of a validation of the JWT provided from the IDP is done easily via the restrict policy called validate-jwt

Let’s go thur how the setup looks like, we will need to set up a Trust between your API Management instance and your Auht0 instance.

I’ve created a video that will go thru all of this a link is provided bellow.

Links used in the video:

Validate JWT Token
Auth0 Openid Configuration url: https://YOUR_AUTH0_DOMAIN/.well-known/openid-configuration

Summary:

Adding a second security layer like this increases security and as you will see later on flexibility. It’s an awesome start in order to build a nice consumer experience for your API’s. In API Management it’s very easy to attach any IDP so you can pick and choose your favourite and the setup will be somehwat similar.

Azure Functions logs in Application Insights part 2

2019-08-23T00:00:00+00:00

In part one I covered the ILogger interface and what it’s providing for us, but sometimes we want more control of our logging, enrich oru logging or we have allready implemented alot of logging with TelemtryClient and just want to connect the logging to our end-to-end experience.

We use the same Scenario, a message is recieve via API Management, sent to a Azure Function that publishes the message on a topic, a second Azure Function reads the published message and publishes the message on a second topic, a third Azure Function recieves the message and sends the message over http to a Logic App that represents a “3:e party system”.

Read more about the scenario background and the standard ILogger interface in previous post Azure Functions logs in Application Insights.

Connecting custom logs from TelemetryClient

If we need more custom logging options than the provided in the standard ILogger provides or we have alot of logging today with TelemetryClient that is not connected to the endo to end scenario. We need attach these to our end-to-end logging experience via the TelemtryClient to take advantage of the more advanced logging features.

In order to achieve this we need to add some context properties to our instance, here is one way to do this via Service Bus, here we recieve the Message object.

From the Message object we can extract the Activity, this is an extension found in the Microsoft.Azure.ServiceBus.Diagnostics namespace. From the result of this operation we can get the RootId and ParentId wich is the values for the current Operation Id and the parent id. Read more on how the Operation id is built and how the hierarchy is designed

public string Run([ServiceBusTrigger("topic1", "one", Connection = "servicebusConnection")]Message message, ILogger log)
{
    string response = "";
    var activity = message.ExtractActivity();

    telemetryClient.Context.Operation.Id = activity.RootId;
    telemetryClient.Context.Operation.ParentId = activity.ParentId;

    telemetryClient.TrackTrace("Received message");

This can also be done from a HTTP call read more.

// If there is a Request-Id received from the upstream service, set the telemetry context accordingly.
if (context.Request.Headers.ContainsKey("Request-Id"))
{
    var requestId = context.Request.Headers.Get("Request-Id");
    // Get the operation ID from the Request-Id (if you follow the HTTP Protocol for Correlation).
    telemetryClient.Context.Operation.Id = GetOperationId(requestId);
    telemetryClient.Context.Operation.ParentId = requestId;
}
	
//GetOperationId method
public static string GetOperationId(string id)
{
    // Returns the root ID from the '|' to the first '.' if any.
    int rootEnd = id.IndexOf('.');
    if (rootEnd < 0)
        rootEnd = id.Length;

    int rootStart = id[0] == '|' ? 1 : 0;
    return id.Substring(rootStart, rootEnd - rootStart);
}

Now we will also add this small line of code that trackes an event.

var evt = new EventTelemetry("Function called");
evt.Context.User.Id = "dummyUser";
telemetryClient.TrackEvent(evt);

And this can now be found in the end-to-end tracing.

Adding an Extra “Operation”

This is probobly my favorit thing in all of this, when processing messages some parts of the process are bigger and/or more important to understand, this could be a complex transformation or just a set of steps that we want to group. Then we can use the StartOperation in TelemetryClient, this will start the scope for the Operation and it will be open until we execute the operation StopOperation it has a status and we can set some properties and get execution time of the process.

An Operation is either a DependencyOperation, intended to mark dependency to other resources or a RequestOperation meant for marking a creation of a request. I will use a DependencyOperation in my case since it’s the most alike operation for my purpose, I use it to mark that a more complex logic has been executed in my function and what the result of it was.

I use this in several ways one is when I process alot of items in my Function to mark that they are executed, like processing a record in a list, another is when performing a complex transformation and so on.

In my example bellow I’ll demonstrate the usage of an operation by doing a simple for loop and simulating processing of a line and adding some valuable metadata. As a tip set the Data property to the object that is going to be processed to easier understand what data is processed.

for (int i = 0; i < 10; i++)
{
    var op = telemetryClient.StartOperation<DependencyTelemetry>(string.Format("Function2_line_{0}", i.ToString()));

    op.Telemetry.Data = string.Format("Processing {0} and random guid {1}", i.ToString(), Guid.NewGuid().ToString());
    op.Telemetry.Type = "LineProcessor";
    op.Telemetry.Success = i < 9;
    op.Telemetry.Sequence = i.ToString();
    op.Telemetry.Context.GlobalProperties.Add("LineID", i.ToString());
	
	//do some heavy work!
	
    telemetryClient.StopOperation(op);                    
}

This will result in a mutch more detailed log in the End-To-End overview

Custom properties is easily set on each operation for traceability or information, bellow is how we can add LineID as a custom property.

op.Telemetry.Context.GlobalProperties.Add("LineID", i.ToString());

We can also set the Operation to have failed even if we continued the run, good for highlight parts of failure while still needing to complete the rest of the run.

Then we can find it under failures and from there start working on how to fix it.

Summary:

In this second part we have covered more advanced logging with TelemetryClient or to connect your current TelemetryClient logging to the End-To-End experience provided by Application Insights.

The StartOperation is really powerfull to highlight what is happening, but I would like to have a third option for an Operation since using the Dependency sounds wrong when it’s a section of the code in the same process, but it works!

This post is about how things can be done and hopefully this can be a guidance in creating a better logging experience, we want to avoid the developers “let me just attach my debugger” response to what is the problem with the flow running in production.

Azure Functions logs in Application Insights

2019-08-19T00:00:00+00:00

Azure Functions is a great tool in our toolbox and as all our tools they have their strengths and flaws. When it comes to logging and monitoring Functions rely on Application Insight’s and later on Azure Monitor. It’s also alot dependent on how the implement of your solution, and there are som out of box features that are really amazing.

If you are unfamiliar with Functions and Application Insights, read more here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-monitoring

By just enabling application insights, without any further addons we get all the logs that are sent to the ILogger object to application Insights. And we also get some End-To-End tracking via supported SDK’s and HttpClient requests. This is very useful to get an understanding of what is happening. So let’s look in to how this manifests in a rather complex scenario.

The Scenario is that a message is recieve via API Management, sent to a Azure Function that publishes the message on a topic, a second Azure Function reads the published message and publishes the message on a second topic, a third Azure Function recieves the message and sends the message over http to a Logic App that represents a “3:e party system”.

After connecting all functions to the same Application Insight instance we are sending a message! Let’s see how this can look like in Application Insights, first we go to the Performance tab:

First let’s just point out some small good to know things, the Operation Name in this view is actually the name of the function that you execute, so Function 1 in the list of Operation Names bellow:

Has this signature in the code FunctionName(“Function1”), this is important as your instances grow make sure the name of the function is unique to easily find em.

 [FunctionName("Function1")]
        [return: ServiceBus("topic1", Microsoft.Azure.WebJobs.ServiceBus.EntityType.Topic, Connection = "servicebusConnection")]
        public static async Task<string> Run(
            [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
            ILogger log)
        {

Then we pick one of the requests, press the samples and pick one, it loads the end-to-end overivew

We can actually see all the steps from the first function to the last function and the call to the Logic App, impressive! The message is beeing sent over Service bus topics but the correlation is all set and we can follow the message all the way!

So this is truly awesome and alot of standard Properties are set out of box, but none specific to our request so it’s hard to understand what data is sent so let’s look in some special ways of adding custom information to the logs.

Logs in Application Insights

Out of box logs is sent to Application Insights via the ILogger object so let’s take a short look at the interface, it contains 3 different operations, Log that logs some information, there are different types and extensions on this to make it easier to use. An isEnabled check to see if the logger is enabled and last a BeginScope operation that will create a scope for our log entries. Read more

public interface ILogger
{
    //
    // Summary:
    //     Begins a logical operation scope.
    //
    // Parameters:
    //   state:
    //     The identifier for the scope.
    //
    // Returns:
    //     An IDisposable that ends the logical operation scope on dispose.
    IDisposable BeginScope<TState>(TState state);
    //
    // Summary:
    //     Checks if the given logLevel is enabled.
    //
    // Parameters:
    //   logLevel:
    //     level to be checked.
    //
    // Returns:
    //     true if enabled.
    bool IsEnabled(LogLevel logLevel);
    //
    // Summary:
    //     Writes a log entry.
    //
    // Parameters:
    //   logLevel:
    //     Entry will be written on this level.
    //
    //   eventId:
    //     Id of the event.
    //
    //   state:
    //     The entry to be written. Can be also an object.
    //
    //   exception:
    //     The exception related to this entry.
    //
    //   formatter:
    //     Function to create a string message of the state and exception.
    void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter);
}

Let’s look at the simplest one just adding a Log text with LogInformation (an extension on Log)some unique data this is to log our productnumber, so we can see what productnumber this request is processing:

	log.LogInformation("ProductNumber {productnumber}", productnumber);

Let’s move to the collected Telemetry to see this log in Azure Portal:

And now we can find our log row:

So now we know this request is processing the specific product, this is searchable in the Search Menu, and an easy click will take you to the End-to-End tracking.

Create a log scope with BeginScope

Sometimes we need to add some data that will be added as properties to all upcoming logs, this can be done via the BeginScope, with the following line all upcoming logs will automatically have the market and productnumber added as properties, the are added as with a prefix of prop__

	using (log.BeginScope("Function3 processed message for {market} with productnumber {productnumber}", market, productnumber))
	{
		log.LogInformation("Processing Lines");
	}

Upcoming Log entry will have prop_market and prop_productnumber.

One good use is to do looping over multiple items, and therefore all entries will have these properties added, also all other properties added with the {} format in log.Information as lineNumber and guid in bellow sample will be added in the customDimension with prefix prop__

	 log.LogInformation("Processing {lineNumber} and random guid {guid}",i.ToString(),Guid.NewGuid().ToString());

And if we want to loop over multiple lines we can add an extra scope to make sure we log the specific unique data for each iteration, following we log the lineNumber and a random guid and as you can see we still have the productnumber and market from before all with the prop__ prefix.

Code sample of the loop

	using (log.BeginScope("Function3 processed message for {market} with productnumber {productnumber}", market, productnumber))
    {
	    log.LogInformation("Processing Lines");
	    for (int i = 0; i < 10; i++)
	    {
	        using (log.BeginScope("Function3 Processing line {lineNumber}", i.ToString()))
	        {
	            log.LogInformation("Processing {lineNumber} and random guid {guid}",i.ToString(),Guid.NewGuid().ToString());
	        }
	    }
	}

Add Filterable Property to Request

Let’s say that we want to not just find a specific run, we want to understand overall performance or errors on a more wider term, like for a market. Let’s consider we want to understand performance or failures for a specific market. This is done via the Performance or Failures tabs, but in order to narrow down the data presented we need to be able to filter the data like the the image bellow adding a filter on Market.

To get this working we need to add an property to our request log. (The request log is the first log event, see the telemetry for more information) the code to achieve this is super simple we just need to add this small snippet of code:

System.Diagnostics.Activity.Current?.AddTag("Market", market);

The function now looks like:

  public static class Function1
    {
        [FunctionName("Function1")]
        [return: ServiceBus("topic1", Microsoft.Azure.WebJobs.ServiceBus.EntityType.Topic, Connection = "servicebusConnection")]
        public static async Task<string> Run(
            [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            dynamic data = JsonConvert.DeserializeObject(requestBody);
            string productnumber = (string)data.productnumber;
            string market = (string)data.marketnumber;

            log.LogInformation("ProductNumber {productnumber}", productnumber);
            System.Diagnostics.Activity.Current?.AddTag("Market", market);

           
            return JsonConvert.SerializeObject(data);
        }
    }

This will now give us the following result:

In the Search section we can pick the Market in the filter to build our queries.

In the Search section we can pick the Market in the filter to build our queries.

In the Performance and Failures sections we need to add a bit more to be able to work with the filter, we need to add customDimensions before Market making the filter look like customDimensions.Market.

And as you can see we get the filtered result after pressing the green confirm button.

Adding more Tags will allow more possibliites for search, the tag will be attached only to the request log entry and not to the following traces. But I would recomend logging both a id and a more broader value like market or similar to find errors/performace issues that span a broader range.

Summary:

I love developing in Azure Functions but in order to have a good Operations experience there is alot of things to consider in the logging strategy to make sure it’s easy to understand what is going on and how to take advantage of the statistics provided by Application Insights.

This post is about how things can be done and hopefully this can be a guidance in creating a better logging and searching experience, we want to avoid the developers “let me just attach my debugger” response to what is the problem with the flow running in production.

August 2019 update Logic App Template Creator

2019-08-12T00:00:00+00:00

There has been some time since last updates where announced on the Logic App Template Creator, but work has been ongoing thanks to contributors so time to form out the new updates.

There are some small changes and bug fixes as usual and fixed some issues reported but here is some new functionality.

The github repository is found here: https://github.com/jeffhollan/LogicAppTemplateCreator

Updates

ISE support, you can now extract Logic Apps in ISE for deployments
Extract custom Connector, you can now extract an ARM template for Custom Connectors
Extract Integration Account Schemas, you can now extract an ARM template for Integration Account Schemas.
Get the Deployed Logic App’s URL as an output parameter from the ARM template deploy (usable when URL is needed in nested ARM templates)
Improved Get-Help method to provide more valuable help and information
Extract Logic App that will be in disabled mode when deployed
Possibility to improved security around parameters and passwords generated in the ARM template and paramter files

Thanks to contributors!

Resource Group Deployments 800 Limit fix

2019-02-12T00:00:00+00:00

There is a limit of how many historical deployments that are saved for a resource group in Azure. Currently the limit is 800 and yet there are no out of box solution to start deleting these historical deployments when the limit is closing in. So what happens is that the next deployment, the 801:st is failing error code is DeploymentQuotaExceeded and message as follows:

Creating the deployment 'cra3000_website-20190131-150417-7d5f' would exceed the quota of '800'. The current deployment count is '800', please delete some deployments before creating a new one. Please see https://aka.ms/arm-deploy for usage details.

So to solve this we manually entered the resource group and deleted some deployments, but that isent what we want to do in the long run. So I looked around and found some powershell scripts I could reuse for this purpose and created a runbbook in an automation account.

I’m not no pro on automation account but I can set em up and use them, there might bsome of you who would give me a tip or 2 on this subject.

So when creating the account we need to use the Azure Run As account, since this will create an account i AAD and this will be used for access, by default the account will have contributor access on the whole subscription (if you have the permissions to create it) otherwise you need someone who can.

If we then look at the resource group we can find this account with contributor access, wich also means that if you only want it to be for sertain resource groups all you need to do is to restrict access for this user and when the script runs Get-AzureRmResourceGroup only the resource groups where this user has access to is returned. (by default the whole subscription)

So the script is simple and there might be some improvements with parallel actions needed for big environments but after running it on schedule for a while it shouldnt be a big issue.

All in all it collects all resource groups that the user has access to and start collecting deployments, in this script I always skip to run the script if there is less than 100 deployments made since it’s no pain in keeping these, it’s just painfull if we get over 800. But if we have more the script starts deleting deployments older than 90 days (by degault) to keep some history there, this can easily be changed in the script. This is run via a Powershell Workbook.

$daysBackToKeep = 90

$connectionName = "AzureRunAsConnection"
try
{
    # Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection = Get-AutomationConnection -Name $connectionName  

    "Logging in to Azure..."
    (Add-AzureRmAccount `
        -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint ) | out-null

    "Obtaining Subscription information..."
    $context = Get-AzureRmContext
    Write-Output ("Subscription Name: '{0}'" -f $context.SubscriptionName)
}
catch {
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    } else {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}

$ResourceGroups = Get-AzureRmResourceGroup

if ($ResourceGroupName)
{
    $ResourceGroups = $ResourceGroups | where { $_.ResourceGroupName -EQ $ResourceGroupName }
}

foreach( $resouregroup in $ResourceGroups) {
    $rgname = $resouregroup.ResourceGroupName
    Write-Output ("Resource Group: '{0}'" -f $rgname)

    $deployments = Get-AzureRmResourceGroupDeployment -ResourceGroupName $rgname
    Write-Output ("Deployments: '{0}'" -f $deployments.Count)

    if($deployments.Count -gt 100 ) {
        $deploymentsToDelete = $deployments | where { $_.Timestamp -lt ((get-date).AddDays(-1*$daysBackToKeep)) }
        Write-Output ("Deloyments to delete: '{0}'" -f $deploymentsToDelete.Count )
        foreach ($deployment in $deploymentsToDelete) { 

            Remove-AzureRmResourceGroupDeployment -ResourceGroupName $rgname -DeploymentName $deployment.DeploymentName -Force
            Write-Output ("Deloyments deleted: '{0}'" -f $deployment.DeploymentName )
        }
        Write-Output ("Deployments deleted")
    }
    else {
        Write-Output ("Less than 100 deployments to deleted, only {0}" -f $deployments.Count)
    }

}

Running it will start deleting deployments, so here is a pre run deployments history

And after we can see that history is deleted (for this run I just removed the 100 check)

After running it the logs will look something like this:

![Runbook logs](/assets/uploads/2019/02/runbook_logs.png

Summary:

This limit is annoying and as I understand something that should be fixed over time, but until then we need to have these kind of helping scripts in environments with alot of deployments. So hopefully this helps out in the meantime.

Fixing Function v2 add extension issues in portal

2019-01-07T00:00:00+00:00

I’m out doing educations and it’s both fun and a great chance for me to learn new stuffs since it’s always something that just for no reason going south.

So today I was out on one of these missions and we where doing some labs developing functions, when we added an extra input from a table storage the Function App just chrashed and then we just got errors that the site was offline and it dident come up. Everybody encountered it so I had to investigate.

The response was a 503 Service Unavailable and the body was the following html content (removed the style since it just took alot of space):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head>
    <title>Host Offline</title>
 </head>
<body>
    <div>&nbsp;</div>
    <div id="wrapper">
        
        <div id="page">
            <div id="content">

                <div class="box">
                    <h2>This Azure Functions app is down for maintenance</h2>
                    <p>
                        Host is offline
                    </p>
                </div>
            </div>
        </div>
    </div>
     
</body>
</html>

So what did we do to get here? Well we just wanted to add an input to from a Table Storage to a standard HTTP Trigger and we started from a blank new defualt function.

Here is my default http function:

Then I went to the Integrate pane and started to add the input from my Azure Table Storage

After selecting the Azure Table Storage we get an warning that the extension is not installed, let’s install it!

It’s installing and we proceed as told in the message:

So we just continued and filled the other details and pressed Save.

After that we went back to the function to add the input, it’s just a simple signature update from:

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)

To:

public static async Task<IActionResult> Run(HttpRequest req,Newtonsoft.Json.Linq.JArray inputTable, ILogger log)

Now when we run the function the inputTable should be populated from the Azure Table Storage but when we try to execute it we just get a http error code 503 Service Unavailable and in the body I can see the text “Host Offline”.

So we started to look into this and read that it could be due to extension not installed properly, to fix this we followed this guide: https://github.com/Azure/azure-functions-host/wiki/Updating-your-function-app-extensions

If you have the same problemes make sure that the extension file exists and contains atleast “Microsoft.Azure.WebJobs.Extensions.Storage if not create and/or add “Microsoft.Azure.WebJobs.Extensions.Storage to the file. Sample bellow:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <WarningsAsErrors />
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.0.*" />
  </ItemGroup>
</Project>  

If you are missing the file it can easily be created in the command prompt witb a simple script:

echo.>extensions.csproj

Run the build command (this takes a while to run so be patient):

dotnet build extensions.csproj -o bin --no-incremental --packages D:\home\.nuget

So now the extensions is installed, let’s verify if this solves the issue completly? Start the Function App and test the function again.

If you are as unlucky as me and the function is not working we need to do one more thing. Let’s go back to Kudos and the CMD Prompt again, the problem is the app_offline.htm file:

Delete it and go back and test agian, it works again!

Summary:

The problem is that the extension is not properly installed and when we add the extension in the configuration without the extension installed the Function App chrashes as produces the app_offline.htm and if it’s present in the folder the response is defaulted to 503 and Host Offline, by removing the file the Function App starts executing as normal and if we have fixed the extensions no errors comes up. This works for all extensions.

By showing the problem and the reproduce scenario we can help out improving the products! So I hope this helps anyone and leads to a fix from the prodcut team.

Logic App Template Creator Updates Januari 2019

2019-01-02T00:00:00+00:00

Updates on the Logic Apps Template Creator has been published:

A minor thing for usage but great for consistency and quality in the generator is that there is now a build and release pipeline setup in DevOps.

Added Build & Release via DevOps to increase quality in merged sprints and automate release to PowerShell Gallery LogicAppTemplate
Improved support for Connectors to Storage Tables and Queues.
Added Commandlet to generate ARM Template for Integration Account Maps

Power Shell Gallery

Now the LogicAppTemplate is updated more frequqntly since I’ve added Build and Release setup to publish new releases to PowerShell Gallery.

PS> Install-Module -Name LogicAppTemplate

Or update to the newest version

PS> Update-Module -Name LogicAppTemplate

Storage Connector Tables and Queues

Now generated on the same way as Storage Blob Connector, meaning that the key will be collected duirng the deployment time based on the storage account name instead of needed to be provided as parameters. This will make it simpler and more neat to do deployments.

There are 3 parameters added

"azureblob_name": {
      "type": "string",
      "defaultValue": "azureblob"
    },
    "azureblob_displayName": {
      "type": "string",
      "defaultValue": "myDisplayName"
    },
    "azureblob_accountName": {
      "type": "string",
      "defaultValue": "myStorageAccountName",
      "metadata": {
        "description": "Name of the storage account the connector should use."
      }
    }

And they are later used in the connection to get the accountKey automatically during deployment.

 {
      "type": "Microsoft.Web/connections",
      "apiVersion": "2016-06-01",
      "location": "[parameters('logicAppLocation')]",
      "name": "[parameters('azureblob_name')]",
      "properties": {
        "api": {
          "id": "[concat('/subscriptions/',subscription().subscriptionId,'/providers/Microsoft.Web/locations/',parameters('logicAppLocation'),'/managedApis/azureblob')]"
        },
        "displayName": "[parameters('azureblob_displayName')]",
        "parameterValues": {
          "accountName": "[parameters('azureblob_accountName')]",
          "accessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('azureblob_accountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]"
        }
      }
    }

All magic is in this ListKeys part, so this tells ARM to collect the key based on a reference to the storage account. (this also means that the account doing the Resource Group Deployment also needs access to the storage account).

[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('azureblob_accountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]

New CommandLet Get-IntegrationAccountMapTemplate

So another improvement is that we now can extract a map from an Integration Account in to a directly deployable ARM template. It work similar to the other extractions and bellow is a sample of how to get the Integration Account map as an ARM template.

Get-IntegrationAccountMapTemplate -ArtifactName 'mapname' -IntegrationAccount 'ianame' -ResourceGroup 'myresourcegroup' -SubscriptionId 'guid()' -TenantName 'mattiaslogdberg.onmicrosoft.com'

Summary:

Exporting Logic Apps via the Logic App Template Extractor simplifies the CI/CD area of using Logic Apps. Without needing to manually add extra work, making the development inside the portal and then just extracting the work and setup the import to the next environment .

API Management Template Creator Updates December 2018

2018-12-21T00:00:00+00:00

Updates on the API Management Template Creator has been published:

I’ve got a lot of help this time frome some awesome contributors a big thanks to them! NilsHedstrom, Geronius, joeyeng, Lfalk.

A minor thing for usage but great for consistency and quality in the generator is that there is now a build and release pipeline setup in DevOps.

Improved support for Identity Providers, Products and more
Added automation for Build & Release via DevOps to increase quality in merged sprints and automate release to PowerShell Gallery
Added to PowerShell Gallery APIManagementTemplate
Split the Template to a template per resource to follow API Management DevOps best pratices

Power Shell Gallery

Now it’s mutch easier to get started, just install the APIMangementTemplate modul from PowerShell Gallery.

PS> Install-Module -Name APIManagementTemplate

Or update to the newest version

PS> Update-Module -Name APIManagementTemplate

Split the Template to a template per resource

In order to follow the best pratice from Azure API Management DevOps example we now have a new command Write-APIManagementTemplates this command will take a template as input and split into a file per resource to make it easy to manage and handle files and create more customized deployment with a linked template. Read more at GitHub

 Get-APIManagementTemplate -APIManagement MyApiManagementInstance -ResourceGroup myResourceGroup -SubscriptionId 80d4fe69-xxxx-4dd2-a938-9250f1c8ab03 | Write-APIManagementTemplates -OutputDirectory C:\temp\templates -SeparatePolicyFile $true

Summary:

Exporting API’s ia the API Management Template Extractor simplifies the CI area of using API Management, here we can select a specific API and export only that API, with Operations, version sets, schemas, properties etc. Without needing to manually add extra work, making the development inside the portal and then just extracting the work and setup the import to the next environment .

Innovation enabled by Integration

2018-08-31T00:00:00+00:00

Normally we focus on technical implementations and problem solving but from time to time we get the opportunity to expand the view and focus on innovative thinking. This is the time where we can show our customers the real value of applying good architecture and principles. Currently I’m having so much fun doing innovative prototyping work with one of my customers that I need to share it with you. We are using Power Apps and Flow for our prototyping in order to get prototypes done fast and easy, without spending too much time and effort.

Power Apps is a platform where you easy can build apps in a point-and-click approach and to multiple device types at the same time, read more at: https://powerapps.microsoft.com/

Flow is a lightweight integration platform that runs on top of Logic Apps, it’s designed to make it easy to create your own flows and turn repetitive tasks into automated workflows, read more at https://emea.flow.microsoft.com/

These platforms are very easy to get started with and there are several samples of applications regarding office365 and common applications. But the real power for innovation comes clear if you have a good API landscape that could be used when building these applications.

We have a customer that we started to work with for almost a year ago and now when all the hard work is done, and we have a good integration landscape we can harvest the benefits of our good architecture. I started with showing the case described below and now they really hunger for more of this kind of innovation both for internal and external use. It’s so wonderful to see how our hard work pays off. Showing that we really are the “integration heroes”, we are the enablers, making this possible.

Architecture that enables innovation?

In reality it’s quite simple, create a platform that contains API’s and flows that can be reused. Make sure to build domain model-based API’s and function driven API’s and focus on making them simple to use and most important reusable. Here is a Sample Architecture:

And in our API Management instance we have created a few domain model API’s:

A few months back we helped in replacing an internal system for product information, it was a system where you could enter a serial number and get basic information about the product. The end solution was a website that used a couple of API’s to retrieve the information needed. The fun part of this was that most of the API’s where already created in other projects for other/similar needs but used from other systems/apps. All we did was fine tune an API, leaving us with time to innovate. On a computer it makes sense to write a serial number but on a mobile phone we expect to be able to scan the serial. Since we had the time we started to investigate the problems around this and if this was solvable.

We started to look in to this and the first “problem” was that the serial plates on the machines had no barcodes, only pure text:

Luckily this function exists in the Cognitive services in Azure read more at Azure Cogniative Services

We started to test this with a Power App that used a Flow to send the image to the Cognitive Service. The Power App solution is very simple, it uses the camera to take a picture and afterwards sends the picture to a Flow that utilizes the cognitive service to identify the serial number from the plate. The result from the Flow is a URL with the serial number that we used to launch the website, making use of the web app. With this we had expanded the functionality with a Serial Plate Scanner and provided basis for the decision on investing in a mobile app.

Here is the flow:

Here is the Power App in the Power Apps Studio developed in my browser:

As you can see there is not much going on but a Camera item with a OnSelect statement that launches the web browser with the URL that comes back from our Flow “Get-Path-To-Product”.

The flow is also quite simple:

The tricky part was how to send the image to the Cognitive Services, in this case we created a compose action with a dataUritoBinary function:

*dataUriToBinary(triggerBody()[‘Skapa_Indata’])

And in the HTTP action we send the binary data to the OCR service: (in the sample below the OCR service is behind our API Management instance)

All in all, the prototype was easy and fast to build, but most importantly it successfully gave us the power to demo the concept. Providing a visual demo to show the value of having an app that can scan the serial plate. This gave a good understanding in the benefits of the investment on a native app to decision makers without spending too much time and resources.

Summary:

The app is currently under development and we have more of these innovative prototypes coming up to prototype new or upgraded functionality in a variety of sizes. Best part of using this approach is that we can create a prototype in a day or two providing fast prototyping, giving decision makers a visual and testable solution to review. All thanks to our hard work with API’s, workflows and the amazing tools Power Apps and Flow!

Become the Integration Hero that you really can be and let that unused innovation power free!

API Management Template Creator Updates August 2018

2018-08-28T00:00:00+00:00

Updates on the API Management Template Creator has been published a few weeks back:

A minor thing for usage but great for consistency and quality in the generator is that there are alot of tests added.

Functions support added
Updated Logic App Support
Removed some name parameter generation

Functions support added

Finally the support for API’s that are generated from Functions is live, now an API generated from Functions get the appropiate automatic generation of function keys via ARM functions. Meaning that you don’t need to supply the function kode in the parameter files they are collected automatically via Listsecrets function in ARM.

Now:

{
      "comments": "Generated for resource /subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/properties/5b41934c6d0f59440d20c5ee",
      "type": "Microsoft.ApiManagement/service/properties",
      "name": "[concat(parameters('service_ibizmalo_name'),'/','5b41934c6d0f59440d20c5ee')]",
      "apiVersion": "2017-03-01",
      "properties": {
        "displayName": "maloapimtest_HttpTriggerAdminKey_query_5b41934c6d0f59440d20c5ee",
        "value": "[listsecrets(resourceId(parameters('FunctionApp_maloapimtest_resourceGroup'),'Microsoft.Web/sites/functions', parameters('FunctionApp_maloapimtest_siteName'), parameters('operations_api-HttpTriggerAdminKey-post_name')),'2015-08-01').key]",
        "tags": [],
        "secret": true
      },
      "resources": [],
      "dependsOn": []
    }

Previously:

parameters: {
    "INT2051functionkey_value": {
      "type": "securestring",
      "defaultValue": "Aze/1W8wCwItuP8JacdyHa2vDw8YScrOkbq6uHcTXiOasb2wi3kZoQ=="
    }
 }
	. . . 
 {
      "comments": "Generated for resource /subscriptions/1fake145-d7f4-4d0f-b406-7394a2b64fb4/resourceGroups/Api-Default-West-Europe/providers/Microsoft.ApiManagement/service/apidev/properties/int2051functionkey",
      "type": "Microsoft.ApiManagement/service/properties",
      "name": "[concat(parameters('service_cpidev_name'), '/' ,parameters('property_INT2051functionkey_name'))]",
      "apiVersion": "2017-03-01",
      "properties": {
        "displayName": "INT2051functionkey",
        "value": "[parameters('INT2051functionkey_value')]",
        "tags": null,
        "secret": true
      },
      "resources": [],
      "dependsOn": []
}

Updated Logic App Support

Two bigger things has been targeted to improve the export and later deployment experience for APIM API’s that uses Logic Apps.

Trigger name fix

A Logic App Trigger can now have any name, previous it was hardcoded to ‘Manual’. There is now logic added to retrieve the information at extraction time, the extractor fetches the Logic App definition and finds the HTTP trigger to get the correct name.

Trigger name is extracted from the Logic App and added to the ARM functions (customtriggername in below sample):

Will result in (note the ‘customtriggername’):

"[listCallbackUrl(resourceId(parameters('LogicApp_customtrigger_resourceGroup'), 'Microsoft.Logic/workflows/triggers', parameters('LogicApp_customtrigger_logicAppName'), 'customtriggername'), '2017-07-01').queries.sig]"

Add Logic App as operation to exisitng API fix

There has been diffrent behavior between API’s generated from a Logic App imported as a new operation to an existing API. The main diffrent has been that the sig has been added as a “normal” parameter and no ARM functionality added to generate the sig. This is now adressed and the appropiate ARM functions will be added and generated automatically for both types.

Properties will get value from the [ListCallBackUrl] function:

{
      "comments": "Generated for resource /subscriptions/1fake145-d7f4-4d0f-b406-7394a2b64fb4/resourceGroups/Api-Default-West-Europe/providers/Microsoft.ApiManagement/service/apidev/properties/int7100-payment-to-o-test_manual-invoke_5b349f2a42974a989226cf33",
      "type": "Microsoft.ApiManagement/service/properties",
      "name": "[concat(parameters('service_apidev_name'), '/' ,parameters('property_int7100-payment-to-o-dev_manual-invoke_5b349f2a42974a989226cf33_name'))]",
      "apiVersion": "2017-03-01",
      "properties": {
        "displayName": "int7100-payment-to-o-dev_manual-invoke_5b349f2a42974a989226cf33",
        "value": "[listCallbackUrl(resourceId(parameters('LogicApp_INT7100-Payment-To-o-DEV_resourceGroup'), 'Microsoft.Logic/workflows/triggers', parameters('LogicApp_INT7100-Payment-To-o-DEV_logicAppName'), 'request'),'2017-07-01').queries.sig]",
        "tags": [],
        "secret": true
      },
      "resources": [],
      "dependsOn": []
    }

Removed some name parameters

Parameters generated for name of resources like, Policy, version sets, operations etc has no real use case or benefits associcated with changing the name and is therefore removed to simplify the ARM templates.

Sample of parameters that are now removed:

    "versionset_5af9817ca656c6952416b779_name": {
      "type": "string",
      "defaultValue": "5af9817ca656c6952416b779"
    },
    "operations_GetSalesOrderById_name": {
      "type": "string",
      "defaultValue": "GetSalesOrderById"
    },
    "operations_policy_name": {
      "type": "string",
      "defaultValue": "policy"
    }

Summary:

API Management Template Creator Updates May 2018

2018-05-14T00:00:00+00:00

Updates on the API Management Template Creator has been dragging but now I’m pleased to have fixed the two most urgent ones.

Support for Versions, with version sets
Schemas

Support for Versions, with Version Sets

If an API that is extracted has enabled the current versionset is also extracted and provided inside the ARM Template. The version set states the version that is used and is needed byt the API to be able to support versions, it contains the information about how the verions is handled for the specific API. The sample bellow is using a HTTP header named API-Version to set the version of the API, the version later is then set on the API itself.

{
      "comments": "Generated for resource /subscriptions/fake439-770d-43f3-9e4a-8b910457a10c/resourceGroups/API/providers/Microsoft.ApiManagement/service/dev/api-version-sets/5ae6f90fc96f5f1090700732",
      "type": "Microsoft.ApiManagement/service/api-version-sets",
      "name": "[concat(parameters('service_dev_name'), '/' ,parameters('versionset_5ae6f90fc96f5f1090700732_name'))]",
      "apiVersion": "2017-03-01",
      "properties": {
        "displayName": "Arkivet",
        "description": null,
        "versioningScheme": "Header",
        "versionQueryName": null,
        "versionHeaderName": "API-Version"
      },
      "resources": [],
      "dependsOn": []
    }

Support for Schema

If an API has schemas added to oeprations these schemas will be extracted and added to the ARM template, all operations will also have a “dependsOn” to the specific schema to prevent errors when executing the ARM template. The sample bellow is a simple schema that is added in the ARM template and referenced in the Operations section.

{
          "comments": "Generated for resource /subscriptions/fake439-770d-43f3-9e4a-8b910457a10c/resourceGroups/API/providers/Microsoft.ApiManagement/service/dev/apis/arkivet/schemas/5af038365b73730dd01453ad",
          "type": "Microsoft.ApiManagement/service/apis/schemas",
          "name": "[concat(parameters('service_dev_name'),'/',parameters('api_arkivet_name'), '/5af038365b73730dd01453ad')]",
          "apiVersion": "2017-03-01",
          "properties": {
            "contentType": "application/vnd.ms-azure-apim.swagger.definitions+json",
            "document": {
              "definitions": {
                "Body": {
                  "type": "object",
                  "properties": {
                    "No": {
                      "type": "string"
                    },
                    "ReportedDate": {
                      "type": "string",
                      "format": "date-time"
                    }
                  }
                },
                "BodyExists": {
                  "type": "object",
                  "properties": {
                    "CivicNo": {
                      "type": "string"
                    }
                  }
                },
                "Post400ApplicationJsonResponse": {
                  "type": "object",
                  "properties": {
                    "message": {
                      "type": "string"
                    },
                    "description": {
                      "type": "string"
                    },
                    "errors": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "properties": {
                          "field": {
                            "type": "string"
                          },
                          "message": {
                            "type": "string"
                          }
                        }
                      }
                    },
                    "hasErrors": {
                      "type": "boolean"
                    }
                  }
                },
                "ExistsPost200ApplicationJsonResponse": {
                  "type": "object",
                  "properties": {
                    "arkivet": {
                      "type": "string"
                    }
                  }
                }
              }
            }
          },
          "resources": [],
          "dependsOn": []
        }

Summary:

Access AAD Secured Web API's from API Management

2018-03-07T00:00:00+00:00

Protecting Web Apps and Web API’s by the built in Authentication and authorization in Azure App Service is a great way to protect resources without adding code to handle the authorization. This means that the site or api is fully secure without the need of implementing it, which is a great example of seperation of concerns. Read more on how it works

What we then need is to access the API and in this case we need to be able to access the API via Management using service principals since the client using our API’s in API Management is outside our organization and should not be bothered by our internal security mechanisms.

The scenario: The API is protected by Azure AD and in order to access it we need to add a valid JWT token to the Auhtorization header in the request, first(1) we need to get the JWT token and then(2) add it in the request to the API App.

In this post we have used the standard generated API App from Visual Studio and published it to a Web API instances in Azure named Protected.

Enabling the built in security Authentication and Authorization to the API’s

In the Azure Portal head to the API App and go in to the settings tab “Authentication/Authorization”. Enable the App Service Authentication by pressing the On button.

Now we can enable several diffrent providers and in this demo we will focus on the Azure Active Directory, press the Active Directory.

After chosing the Azure Active Directory, we are presented with 2 options to configure the Azure Active Directory setup. Express gives some guidance and the possiblity to create or select an existing AD Application while in Advance we can write in the values needed, the AAD Application clientid and the issuer, normaly https://sts.windows.net/{tenantguid}/ followed by the tenat guid URL. In this setup we will use the Express option and create a new AD Application.

Here we can choose to use an existing AD Application or create a new one, I will create a new one named ProtectedAppAAD.

After this is installed we can now save and test that the API is protected. If you open a new browser or log out from the account you should be forced to login and the site is now secured.

So now the site is securely protected and we can access it when logged in, now we need to be able to access the API via API Managamenet.

Provide access to the API

So the API is now protected and we need to provide a way for API Management to access it, we will need to use another AAD Application (App Registration) and provide access to the ProtectedAppAAD. So we need to create a new Application, named APIManagementAADApp, the sign in URL can be http://localhost.

After the AAD application is created we need to provide access to the ProtectedAppAAD this is done via permissions; to assign permissions go to Settings, press Required Permissions and Add.

Under Select an API we need to search for ProtectedAppAAD since first the list only shows Microsoft Standard API’s. Enter the ProtectedAppAAD in the text box and press Select.

Now we need to select the permission and the one we want is Access, select it and press Select, finish the setup by press Done.

Last step is pressing the Grant Permissions to enable the permissions (do not forget this!).

API Management Policy for Aquiring JWT Token

In order to be able to expose this API we need to get a token from AAD using the Application, this will be done inside a policy and luckily for us the API Management team has provided a set of code snippets at GitHub and one of these is doing exactly that, get it here.

There is more of them at the repository in GitHub and if you have a great snippet that you want to share you can add a Pull Request and if the team approves it. it will end up there. Here is the GitHub repository, https://github.com/Azure/api-management-policy-snippets/blob/master/Snippets/.

The Get OAuth2 access token from AAD snipet looks like this.

<!-- The policy defined in this file provides an example of using OAuth2 for authorization between the gateway and a backend. -->
<!-- It shows how to obtain an access token from AAD and forward it to the backend. -->

<!-- Send request to AAD to obtain a bearer token -->
<!-- Parameters: authorizationServer - format https://login.windows.net/TENANT-GUID/oauth2/token -->
<!-- Parameters: scope - a URI encoded scope value -->
<!-- Parameters: clientId - an id obtained during app registration -->
<!-- Parameters: clientSecret - a URL encoded secret, obtained during app registration -->

<!-- Copy the following snippet into the inbound section. -->

<policies>
  <inbound>
    <base />
      <send-request ignore-error="true" timeout="20" response-variable-name="bearerToken" mode="new">
        <set-url></set-url>
        <set-method>POST</set-method>
        <set-header name="Content-Type" exists-action="override">
          <value>application/x-www-form-urlencoded</value>
        </set-header>
        <set-body>
          @{
          return "client_id=&resource=&client_secret=&grant_type=client_credentials";
          }
        </set-body>
      </send-request>

      <set-header name="Authorization" exists-action="override">
        <value>
          @("Bearer " + (String)((IResponse)context.Variables["bearerToken"]).Body.As<JObject>()["access_token"])
	  </value>
      </set-header>

      <!--  Don't expose APIM subscription key to the backend. -->
      <set-header exists-action="delete" name="Ocp-Apim-Subscription-Key"/>
  </inbound>
  <backend>
    <base />
  </backend>
  <outbound>
    <base />
  </outbound>
  <on-error>
    <base />
  </on-error>
</policies>

The way it works is that two things happen before we send the request to the backend.

First a request in the send-request action is made to the AAD tenant and aquiring a token.
The set-header action is adding the token in the Authroization header, by extracting it from the result from the first request.

But before we can add this snippet to our API we need to add a few values to the Named Values section in our API Management instance.

authorizationServer - the tenant URL My tenant ID is 1cb87777-3df4-428b-811a-86a0f215cd35 so the URL then is https://login.microsoftonline.com/1cb87777-3df4-428b-811a-86a0f215cd35/oauth2/token
clientId - the Application id of our AAD Application in this case of APIManagementAADApp
clientSecret - the key to our AAD Application in this case of APIManagementAADApp
- Create a new key
scope - the AAD Application id of the AAD Application that is protecting our resource, in this case the id of ProtectedAppAAD

After adding these the Named Values section of our API Managemet will end up looking like this. Make sure to add these before adding the policy, since the policy canno be saved if the referenced Named Value dosen’t exist.

Now we just need to add the policy to our API and we will be able to access the protected API, after the Policy is saved test it and you should now recieve the result from the API.

Summary:

Todays solutions almost always accesses other resurces and when building this landscape we need to make sure that all parts are protected and secure, often even we need to combine a few of them to be compliant with requirments from customers. In this post I showed how to use one of the built in security from Web Apps with AAD that can be used standalone or as an extra layer of security above the “standard” Basic, Oauth, API keys, Certificates etc. authentications. Another benefit is that is not part of the code so it’s not possible to “forgett” or accidently removed in a bug fix or similar.

As a bonus the API Management Snippets is a really nice initiative with pre created advanced snippets so make sure to check this out.

https://github.com/Azure/api-management-policy-snippets/blob/master/Snippets/

Logic App Custom Connector WSDL Tips

2018-02-02T00:00:00+00:00

So continue my journey with the Logic Apps Custom Connector and this time we are doing some work with the WSDL part of the Logic Apps Custom Connector and I thought I would share some tips and trix I’ve learned and are using when importing and using Connectors with WSDL. Since the only way to be a good user of a resource like the Custom Connector is to understand how we can investigate behaviors, tweak and fix problems and find debug/logs.

Import Error

There are several reasons why there will be errors when importing the WSDL and since it’s API Management functionality we can (until documentation is up to speed) see the limits on the API Management site:

https://docs.microsoft.com/en-us/azure/api-management/api-management-api-import-restrictions#wsdl

I encountered the “rare” error with recursive objects, apparently there where a “lazy” coder that just referenced the whole entity in a parent child situation even if the only “needed” field was the ID. Anyway removing the recursive element solves the problem,alternatively manipulating the representation could have been done.

Runtime problems, unexpected behaviors and errors from the backend system

So there are many reasons why there can be errors sent from the backend system but one of the common ones I’ve found is System.FormatException: Input string was not in a correct format and that is due to a element specified as int and the value sent in is null. Now how can that be a problem? Since the generation is done by the same logic as in API Management we can import the WSDL in API Management and see how the actual liquid template looks like.

Let’s look at an example.

With a system I’m sent a link to the WSDL and a sample this to easy integrate with them, testing the sample works fine but importing the WSDL provides alot more than the sample as we are used to there are more fields than we need. So sending in the test file via postman works good but the connector does not work. Let’s take a look why, bellow is the sample a fairly easy message with few elements.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ProcessArticle xmlns="http://ongoingsystems.se/WSI">
      <GoodsOwnerCode>code1234</GoodsOwnerCode>
      <UserName>user1234</UserName>
      <Password>pass1234</Password>
      <art>
        <ArticleOperation>CreateOrUpdate</ArticleOperation>
        <ArticleIdentification>ArticleNumber</ArticleIdentification>
        <ArticleNumber>6553173735310</ArticleNumber>
		<ArticleName>Display Stand</ArticleName>
        <ArticleDescription>Display Stand</ArticleDescription>
        <ArticleUnitCode>St</ArticleUnitCode>
        <IsStockArticle>1</IsStockArticle>
      </art>
    </ProcessArticle>
  </soap:Body>
</soap:Envelope>

But when using the Custom Connector all the complex structures are mapped: (this is generated via API Management):

 <set-body template="liquid">
			<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns="http://ongoingsystems.se/WSI">
				<soap:Body>
					<ProcessArticle>
						<GoodsOwnerCode></GoodsOwnerCode>
						<UserName></UserName>
						<Password></Password>
						<art>
							<ArticleOperation></ArticleOperation>
							<ArticleIdentification></ArticleIdentification>
							<ArticleSystemId></ArticleSystemId>
							<ArticleNumber></ArticleNumber>
							<ArticleName></ArticleName>
							<ProductCode></ProductCode>
							<BarCode></BarCode>
							<SupplierArticleNumber></SupplierArticleNumber>
							<ArticleDescription></ArticleDescription>
							<ArticleUnitCode></ArticleUnitCode>
							<CountryOfOriginCode></CountryOfOriginCode>
							<StatisticsNumber></StatisticsNumber>
							<PurchaseCurrencyCode></PurchaseCurrencyCode>
							<Weight></Weight>
							<NetWeight></NetWeight>
							<Volume></Volume>
							<Length></Length>
							<Width></Width>
							<Height></Height>
							<QuantityPerPallet></QuantityPerPallet>
							<QuantityPerPackage></QuantityPerPackage>
							<OrderPoint></OrderPoint>
							<Price></Price>
							<CustomerPrice></CustomerPrice>
							<PurchasePrice></PurchasePrice>
							<IsStockArticle></IsStockArticle>
							<ArticleGroup>
								<ArticleGroupOperation></ArticleGroupOperation>
								<ArticleGroupIdentification></ArticleGroupIdentification>
								<ArticleGroupCode></ArticleGroupCode>
								<ArticleGroupName></ArticleGroupName>
							</ArticleGroup>
							<ArticleCategory>
								<TypeOperation></TypeOperation>
								<TypeIdentification></TypeIdentification>
								<Code></Code>
								<Name></Name>
							</ArticleCategory>
							<VatCode>
								<VatCodeOperation></VatCodeOperation>
								<VatCodeIdentification></VatCodeIdentification>
								<VatCode></VatCode>
								<VatPercent></VatPercent>
							</VatCode>
							<DangerousGoods>
								<UNNumber></UNNumber>
								<UNIsMarineHazard></UNIsMarineHazard>
								<UNIsDangerousGoods></UNIsDangerousGoods>
								<UNPackageType></UNPackageType>
								<UNTunnelCodes>
</UNTunnelCodes>
								<UNClassNumber></UNClassNumber>
								<UNProperShippingName>
									<Name></Name>
									<LanguageCode></LanguageCode>
								</UNProperShippingName>
								<UNLabelNumbers></UNLabelNumbers>
								<DangerousGoodsCoefficient></DangerousGoodsCoefficient>
								<EmSCode></EmSCode>
							</DangerousGoods>
							<ArticleNames>
</ArticleNames>
							<ArticleStructureSpecification>
</ArticleStructureSpecification>
							<MainSupplier>
								<SupplierIdentificationType></SupplierIdentificationType>
								<SupplierOperation></SupplierOperation>
								<SupplierNumber></SupplierNumber>
								<SupplierName></SupplierName>
								<Address>
									<Name></Name>
									<Address></Address>
									<Address2></Address2>
									<Address3></Address3>
									<PostCode></PostCode>
									<City></City>
									<TelePhone></TelePhone>
									<Remark></Remark>
									<Email></Email>
									<MobilePhone></MobilePhone>
									<IsEuCountry></IsEuCountry>
									<CountryStateCode></CountryStateCode>
									<CountryCode></CountryCode>
									<DeliveryInstruction></DeliveryInstruction>
									<IsVisible></IsVisible>
									<NotifyBySMS></NotifyBySMS>
									<NotifyByEmail></NotifyByEmail>
									<NotifyByTelephone></NotifyByTelephone>
								</Address>
								<comment></comment>
							</MainSupplier>
							<IsGSPCertified></IsGSPCertified>
							<MaxStockDays></MaxStockDays>
							<BarCodePackage></BarCodePackage>
							<LinkToPicture></LinkToPicture>
							<BarCodePallet></BarCodePallet>
							<QuantityPerLayer></QuantityPerLayer>
							<PalletHeight></PalletHeight>
							<TaricNumbers>
</TaricNumbers>
							<IsObsolete></IsObsolete>
							<MinDaysToExpiryDate></MinDaysToExpiryDate>
							<AdditionalStatisticsNumber></AdditionalStatisticsNumber>
							<CustomsExportConditions></CustomsExportConditions>
							<ArticleColor>
								<ColorCode></ColorCode>
								<ColorName></ColorName>
							</ArticleColor>
							<ArticleSize>
								<SizeCode></SizeCode>
								<SizeName></SizeName>
							</ArticleSize>
							<IsSerialNumberArticle></IsSerialNumberArticle>
							<IsBatchArticle></IsBatchArticle>
							<ArticleDefinitionClasses>
								<ArticleDefinitionClassesOperation></ArticleDefinitionClassesOperation>
								<Classes>
</Classes>
							</ArticleDefinitionClasses>
							<ArticleFreeDecimal1></ArticleFreeDecimal1>
							<ArticleFreeDecimal2></ArticleFreeDecimal2>
						</art>
					</ProcessArticle>
				</soap:Body>
			</soap:Envelope>
		</set-body>

And we start running in to problems due to the fact that we are not sending in more data than earlier but the generated xml is much larger and just look at the enormous representation in the GUI:

Therefore we get some unwanted errors since there are some implications added when sending in starts kin complex structures, since I can’t really do much about the Logic I need to modify the data sent in, but I can modify the WSDL and reimport it. So after changing the WSDL to only contain the elements that I needed in my request it looks alot better and the xml message sent are now matching the sample:

Modified definition in the WSDL so the definition is minimal:

<s:complexType name="ArticleDefinition">
        <s:sequence>
          <s:element minOccurs="1" maxOccurs="1" name="ArticleOperation" type="tns:ArticleOperation" />
          <s:element minOccurs="1" maxOccurs="1" name="ArticleIdentification" type="tns:ArticleIdentificationType" />
          <s:element minOccurs="0" maxOccurs="1" name="ArticleNumber" type="s:string" />
          <s:element minOccurs="0" maxOccurs="1" name="ArticleName" type="s:string" />
          <s:element minOccurs="0" maxOccurs="1" name="ArticleDescription" type="s:string" />
          <s:element minOccurs="0" maxOccurs="1" name="ArticleUnitCode" type="s:string" />
          <s:element minOccurs="1" maxOccurs="1" name="IsStockArticle" nillable="true" type="s:boolean" />
          <s:element minOccurs="0" maxOccurs="1" name="Weight" type="s:decimal" />
          <s:element minOccurs="0" maxOccurs="1" name="NetWeight" type="s:decimal" />
          <s:element minOccurs="0" maxOccurs="1" name="Volume" type="s:decimal" />
          <s:element minOccurs="0" maxOccurs="1" name="Length" type="s:decimal" />
          <s:element minOccurs="0" maxOccurs="1" name="Width" type="s:decimal" />
          <s:element minOccurs="0" maxOccurs="1" name="Height" type="s:decimal" />
          <s:element minOccurs="0" maxOccurs="1" name="QuantityPerPallet" type="s:int" />
          <s:element minOccurs="0" maxOccurs="1" name="QuantityPerPackage" type="s:int" />
        </s:sequence>
      </s:complexType>

Sample from the GUI:

The request can now be sent to the backend without any problems. This approach can be used to both detect problems and also understand the behavior of the Custom Connector and changeing the WSDL can help us to easier use the Connector even if the maintainance is heavier and we need to keep track of these changes and do the again if a update WSDL would be imported.

Summary

As stated we need to find the ways of understanding the resource before beeing good users of it, so I hope this will help out and give ideas around workarounds for some troublesome scenarios and that more debug and customization properties is coming along the way. When more power is needed I would advise using API Management for now until there is more customization properties but most of the time it works like a charm!

CI with Logic App Custom Connector

2018-01-29T00:00:00+00:00

Just had some fortune to work with some projects using the Logic App Custom Connector it’s alot of fun now with all new capabilites on prem connectivity and all and as allways we end upp with setting up CI deployments via TFS.

That is when I found that the On Prem Data Gateway ARM template is poorly documented and the generated template from automation script are missing some important properties. So I thought I’ll gather the information needed here, provide som pitfalls and so on.

The schema for the ARM template can be found at GitHub: https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2016-06-01/Microsoft.Web.json.

It’s inside the Web schema and not that “easy” to get that you are in the correct definition but go down to row 119 and you find the customApis part, this is good for reference.

So let’s see how a ARM template for the On Premise Data Gateway looks like when complete: ( I’m actually adding a fully working connector for https://billogram.com/) The full represenatation of the swagger can also be found at my GitHub repository https://github.com/MLogdberg/LogicAppCustomConnectors/tree/master/Billogram

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "customApis_name": {
      "defaultValue": "Billogram",
      "type": "String"
    },
    "location": {
      "defaultValue": "westeurope",
      "type": "String"
    },
    "serviceUrl": {
      "defaultValue": "https://sandbox.billogram.com/api/v2",
      "type": "String"
    }
  },
  "variables": {},
  "resources": [
    {
      "type": "Microsoft.Web/customApis",
      "name": "[parameters('customApis_name')]",
      "apiVersion": "2016-06-01",
      "location": "[parameters('location')]",
      "properties": {
        "connectionParameters": {
          "username": {
            "type": "securestring",
            "uiDefinition": {
              "displayName": "API USer",
              "description": "The API User for this api",
              "tooltip": "Provide the API User",
              "constraints": {
                "tabIndex": 2,
                "clearText": true,
                "required": "true"
              }
            }
          },
          "password": {
            "type": "securestring",
            "uiDefinition": {
              "displayName": "API Password",
              "description": "The API Password for this api",
              "tooltip": "Provide the API Apssword",
              "constraints": {
                "tabIndex": 3,
                "clearText": false,
                "required": "true"
              }
            }
          }
        },
        "backendService": {
          "serviceUrl": "[parameters('serviceUrl')]"
        },
        "swagger": {
          "swagger": "2.0",
          "info": {
            "description": "The Billogram API is built according to RESTful principles, which means it uses HTTP as an application protocol rather than just as a transport layer for a custom protocol, like SOAP does. In other words, HTTP features such as the various request types (GET, PUT, POST, DELETE), response codes (403 Forbidden, 404 Not Found, 500 Internal Server Error) and standard headers (Accept, Authorization, Cache-Control) are a part of the API.",
            "version": "1.0.0",
            "title": "Swagger Invoice/Billogram",
            "termsOfService": "http://swagger.io/terms/",
            "contact": {
              "email": "billogram@billogram.se"
            }
          },
          "host": "sandbox.billogram.com",
          "basePath": "/api/v2",
          "schemes": [
            "https"
          ],
          "consumes": [],
          "produces": [],
          "paths": {
            ...removed for simplicity display...
              }
            }
          },
          "parameters": {},
          "responses": {},
          "securityDefinitions": {
            "basic_auth": {
              "type": "basic"
            }
          },
          "security": [
            {
              "basic_auth": []
            }
          ],
          "tags": [
            {
              "name": "Billogram Invoice",
              "description": "Handling Billogram/Invoices",
              "externalDocs": {
                "description": "Billogram structure",
                "url": "https://billogram.com/api/documentation#billogram_structure"
              }
            }
          ],
          "externalDocs": {
            "description": "Find out more about Swagger",
            "url": "http://swagger.io"
          }
        },
        "description": "My ARM deployed Custom Connector",
        "displayName": "[parameters('customApis_name')]",
        "iconUri": "/Content/retail/assets/default-connection-icon.6296199fc1d74aa45ca68e1253762025.2.svg"
      },
      "dependsOn": []
    }
  ]
}

Make sure to add both the swagger and the backendService object since the url specified in the serviceUrl since this is mandator and the URL that the Custom Connector are using during runtime, not the one specified in the swagger, this is good for easier management of dev/test/prod environments.

The Custom Connector is a great addon to Logic Apps and by been able to easily manage and deploy them is crusial, so even if this is great to be able to manage my Custom Connector via ARM I’m still missing the possibility to reference my swagger via a url since that would be most suitable way of deploying and kkeping the ARM template and swagger separated.

Januari update Logic App Template Creator

2018-01-16T00:00:00+00:00

A new year comes with new updates, as you might now there is now a Custom Connector option for Logic Apps and earlier this had some changes in the reource id handling in comparision with the managed api’s. So this has now been updated and fixed.

Also there where some issues regarding names that could not contain . (dot) and ohter special characters that has been fixed, a new way of handlign reource id now prevent problems related to naming restrictions.

The last part added was an option to be able to save the incoming body of the requested resources, this was done to be able to easier to debugg when developing, thanks to this and a Mock resource collector class that can replay the saved files we now can create full test runs that are replaying the whole collection of resources.

The github repository is found here: https://github.com/jeffhollan/LogicAppTemplateCreator

Updates

Now supporting extraction of Logic Apps containg Custom Connectors and generating working importable Connection resources with the Logic App.
Updates to resource id handling, now able to handle resource groups, resouirce names with .(dot) and other special characters that previously was not supported
Added a resource handler to be able to print out the incoming requests, to be able to build complete test suites with full replay of a extraction of a Logic App

Logic App Custom Connector via On Prem Gateway

2017-12-28T00:00:00+00:00

In an ever growing hybrid world the need of exposing services that are hosted On Premise are growing and so are the limitations and requirments aswell, the newest way of exposing HTTP based services in Logic Apps is via the Logic App Custom Connector via the On Premise Data Gateway

So previusly when having the task of calling on-premise firewall protected services from our Logic App’s it wasent possible without firewall openings or other resources to help connecting.

But now it’s possible and let’s solve it with the newly updated On Premise Data Gateway and a Logic App Custom Connector.

Prerequisites

Install the Gateway

First of make sure you have the latest version installed, follow the install instructions found here. .

Rememeber the name of the installed Gateway if you have many.

Create the API

In this scenario I will expose a swagger api endpoint, the simple standard Values Api, that are created default in a new Web App project with type Api App after enabling Swagger we can browse the swagger definition, as follows:

Copy the URL to the swagger defintion, shown in the middle bo in the image above ( http://localhost:51921/swagger/docs/v1 ) and save the content as a text file. (Simplest way is to browse it and copy the swagger definition to a text file)

Create Custom Logic App Connector

Now we are prepared and can start of with creating the Logic App Custom Connector based on the swagger file, note that it’s also possible to use WSDL files as source.

In the Azure Portal create a Logic Apps Custom Connector:

No definition is added during the creation time, sp we will have to edit the Custom Connector once it’s created. Start edit the newly created Custom Connector and let’s add the swagger definition:

Press the Edit button.

Upload the swagger file:

Now the API Connection is generated based on the swagger file, scroll down and make changes as you see them neeed but dont’ fotgett to check the “Connect via on-premises data gateway” checkbox and if needed change the URL’s to match the URL’s to the server exposing the service (I’ll just keep em same since it’s on the same machine for me)

Now press Continue and move on with the process, once you are finished make sure to update the Logic Apps Custom Connector by pressing the check mark at the top:

After the Logic Apps Custom Connector is saved we can now use it in our Logic Apps, so lets’ create a Logic App and start using our newly created Custom Connector. (Make sure to create the Logic App in the same Location as your on-premise data gateway is installed otherwise it wont be available in the Logic App).

Inside the Logic App we can now find our Custom Connector among all the others and if I start searching for it I can easily select the appropiate action I want to use:

After I pick my operation, I just took the Get All I need to create the API Connection that will be connected to my Custom Connector, here is where I can pcik authentication method and also the Gateway I want to use.

Pressing create will create the API Connection and the Logic App is completed for this sample, let’s save the Logic App and test run it, the result is that we will get the “values” returned as promised:

The flow is now completed and since we have installed an On Premise Data Gateway on our webservice we can use the Logic App Custom Connector definition in our calls to our on-premise webservice, when creating the Logic App we are creating a API Connection based on the Logic App Custom Connector that holds credentials and Gateway information to be used when executing the Logic App run and we can now connect to our firewall protected on-premise webservice!

Summary:

I think this is an amasing new feature that we really have wanted for a long time. This enables so mutch simpler exposure of services on-premises and in a secure and reliable way. No need for fireall openings, vpn’s, express routes or reverse proxies in a DMZ, we can just install the gateway and use it as our entry point.

Remember this is still in preview and I’ve encountered some issues when working with SOAP so there are some fixes needed, but it looks good so far!

Centralize secrets in Azure Key Vault

2017-11-27T00:00:00+00:00

When working with usernames, passwords or api keys these need to be stored in a secure and manageble way. Usually I find that these are added to Application Settings and manually handled in several places, this is not a desirable way of working and may look something like this, secrets spread out in all areas with red circle:

The first step is to centralize the Values and there I find that Azure Key Vault Vault is a superb place for storage, we got RBAC support for granular security, making it possible for a developer to access the values via code or when deploying via ARM template but not to see or edit the value. So with this we can make sure that passwords/usernames and other secrets are not spread via email’s, stored in dropbox or other “safe” places when distributed to the developers to add them to the App Settings or store them in parameter files for our ARM templates. It also adds reusability so if the value is needed in a Function and in a Logic App we can make sure it’s the same value that is used and we can manage it in one place.

So let’s have this sample in our KeyVault where sqluser and sqlpassword should be used in both a Function and when creating a Logic App Connector.

Finding the Key vault Resource id:

ARM Deployment Logic App Connector

Using KeyVault with ARM deployment requires that the KeyVault has enabled access for Azure Resource Manager for template demployment and that the principal (AAD Application) that is used when deploying has permissions to access the secrets.

After this secrets are accessible via the ARM Template Parameter file, (only from the parameter file or from a Wrapping ARM Template) here is a sample how it would look like in a parameter file:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "name": {
      "value": "INT001-SQL"
    },
    "location": {
      "value": "West Europe"
    }
    "sqlusername" :{
      "reference": {
        "keyVault": {
          "id": "/subscriptions/fake-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/testenvironment/providers/Microsoft.KeyVault/vaults/ibizmalotest"
        },
        "secretName": "sqluser"
      }
    },
	"sqlpassword" :{
      "reference": {
        "keyVault": {
          "id": "/subscriptions/fake-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/testenvironment/providers/Microsoft.KeyVault/vaults/ibizmalotest"
        },
        "secretName": "sqlpassword"
      }
    }
  }
}

When doing a Resource Group Deployment the value is collected from Key Vault during deployment time and then stored in the API Connection. (Remember deployment time means that if the value is changed a new deployment is needed to reflect the change to the API Connection)

Now secrets are stored in Key Vault and the developer setting this up don’t know the actual values in test/prod just what the secret’s name is. The deployment flow looks as follows below where during deployment the secrets is collected from KeyVault (1) and used when creating/updating the API Connection (2) ) this also means that no secrets are stored in the ARM template parameters file and that is great!

IMPORTANT if the value in Key Vault is changed a new deployment is required to update the API Connection.

Accessing Secrets from Functions/Web Apps or other c# applications

Any c# application can easily use the KeyVault SDK to retrieve secrets from Key Vault during runtime, but we do need some things in order to make this possible, from code we need an AAD Application in order to grant access to the secrets, I will not go through how to create one here but read more here

When it’s created we need to get the Application ID (clientId) and one Key (clientSecret).

Now that we have collected the information needed we also need to make sure that our AAD Application has access to the secrets, that is something we are adding to the Key Vault under the “Access Policies”:

Press the “Add new”, select the principal (AAD Application) in my case the KeyVaultApp, in the Secret Permissions we only want and need Get and no more privileges should be given to the Policy.

Now to start coding we need some nuget packages: *Microsoft.Azure.KeyVault; *Microsoft.IdentityModel.Clients.ActiveDirectory

And a bit of code, sample bellow, for more indept about how to use in Functions read more at Jeff Holan’s blog post:

Code to retrieve a secret thesecrets

public static class KeyVaultFunction
{

    private static string clientID = Environment.GetEnvironmentVariable("clientId", EnvironmentVariableTarget.Process);
    private static string clientSecret = Environment.GetEnvironmentVariable("clientSecret", EnvironmentVariableTarget.Process);       
    private static string keyvaultname = Environment.GetEnvironmentVariable("keyvaultname", EnvironmentVariableTarget.Process);    

    [FunctionName("KeyVaultFunction")]
    public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
    {
        log.Info("C# HTTP trigger function processed a request.");

        string secretUri = $"https://{keyvaultname}.vault.azure.net/secrets/";

        var kv = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetToken));
        log.Info("Token aquired");
        var sqlusernameSecret = await kv.GetSecretAsync(secretUri + "sqluser");
		var sqlpasswordSecret = await kv.GetSecretAsync(secretUri + "sqlpassword");
		
		//do some code here

        return req.CreateResponse(HttpStatusCode.OK, "SQL credentials collected ok");
    }

    public static async Task<string> GetToken(string authority, string resource, string scope)
    {
        var authContext = new AuthenticationContext(authority);
        ClientCredential clientCred = new ClientCredential(clientID, clientSecret);
        AuthenticationResult result = await authContext.AcquireTokenAsync(resource, clientCred);

        if (result == null)
            throw new System.Exception("Failed to obtain the JWT token");

        return result.AccessToken;
    }
}

First the settings are added to the Function App Application Settings:

private static string clientID = Environment.GetEnvironmentVariable("clientId", EnvironmentVariableTarget.Process);
private static string clientSecret = Environment.GetEnvironmentVariable("clientSecret", EnvironmentVariableTarget.Process);       
private static string keyvaultname = Environment.GetEnvironmentVariable("keyvaultname", EnvironmentVariableTarget.Process);
private static string secretname = Environment.GetEnvironmentVariable("secretname", EnvironmentVariableTarget.Process);

To handle the authentication we need a method to send i to the AuthenticationCallback object that takes 3 parameters, authority is the login authroity i.e: “https://login.windows.net/1cb87777-3df4-428b-811a-86a0f215cd35” the resoruce is what resource we are trying to access (KeyVault): “https://vault.azure.net” and scope is the scope we are accessing with normaly empty string.

public static async Task<string> GetToken(string authority, string resource, string scope)
{
    var authContext = new AuthenticationContext(authority);
    ClientCredential clientCred = new ClientCredential(clientID, clientSecret);
    AuthenticationResult result = await authContext.AcquireTokenAsync(resource, clientCred);

    if (result == null)
        throw new System.Exception("Failed to obtain the JWT token");

    return result.AccessToken;
}

Then the code part that is actually collecting the secret is the following, we need the URI to the secret that we compose of the Key Vault name and the secret name, and the GetToken function to send in to the AuthenticationCallback object.

string secretUri = $"https://{keyvaultname}.vault.azure.net/secrets/";
var kv = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(GetToken));
var sqlusernameSecret = await kv.GetSecretAsync(secretUri + "sqluser");

After this setup is completed we now are collecting the secrets from Key Vault during runtime and therefore there are no more secrets stored in application settings or in code and we have reused the same secrets in diffrent areas.

Summary:

I really like to remove username and passwords from code and settings since it’s always a mess to manage and in handling them, also there are a few security aspects to consider as well. Making it possible to get username/password or other secrets from Key Vault inside your Function, Web App and at the same time use them when deploying ARM templates like with Logic Apps API Connections or API Management is really the true win, since we now only need to manage these in one place and can use them wherever needed.

In the end we are creating a solution as the image bellow and this gives us a centralized place for our secrets and we only spreading them if we really need to like to an API Connection and that is done integrated with ARM.

OBS! Make note that if we access Key vault via code an update is reflected emediatly while used via ARM a new deployment is needed in order for the new value to be deployed to i.e. password for a API Connection, making it vital to have good Build and Release processes).

Azure Functions npm install via VSTS

2017-11-01T00:00:00+00:00

When setting upp build and release cycles for diffrent Azure resources we bump in to diffrent challenges, this post will cover the challenge of npm install of node modules in to a Function App when doing deployments.

When we are working with our development enviornment it’s okay to go in and run kudo commands and so on to make sure our Function App has the correct packages installed, but as we are adding speed and agility to our processes our Release pipelines in VSTS has to automate more of these processes. It’s not that it’s hard to go in to Kudo and run the command but it’s a manual step that takes time and knowledge when doing a release, I prefer to have everything prepped so that I know things are working when the Release is installed.

So let’s look in to how we can run Kudu commands from VSTS, since Kudu has a REST API we can use it for these kind of tasks, read more on it here Kudu Rest API

When looking in to the API documentation we easily find a function for executing commands:

POST /api/command
{
    "command": 'echo Hello World',
    "dir": 'site\\repository'
}

This means that we could create a message that looked like this for a npm install of the request package:

{
    "command": "npm install request",
    "dir": "site\\wwwroot"
}

So let’s start figuring out the rest, how to execute this from VSTS?

As good as it’s get this can be executed via PowerShell explained and sampled at the bottom of the API reference (I changed the sample in the following code snippet to execute the command for npm install):

$username = "`$website"
$password = "pwd"
# Note that the $username here should look like `SomeUserName`, and **not** `SomeSite\SomeUserName`
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password)))

$userAgent = "powershell/1.0"
$apiUrl = "https://$functionAppName.scm.azurewebsites.net/api/command"
$command = '{"command": "npm install ' + $npmpackage + '","dir": "site\\wwwroot"}'

Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method POST -Body $command -ContentType "application/json"

The credentials is now needed and that is somewhat messy, but since it’s the same as the deployment credentials we can add a commands to get that information via AzureRmResourceAction command:

$creds = Invoke-AzureRmResourceAction -ResourceGroupName $resourceGroup -ResourceType Microsoft.Web/sites/config `
            -ResourceName $functionAppName/publishingcredentials -Action list -ApiVersion 2015-08-01 -Force

So with this information we can now build a more generic script that will only need three (3) parameters to execute a npm install command on our Function App:

functionAppName: the name of the Function App
resourceGroupName: the name of the resource group that contains the Function App
npmpackage: the npm package to install

param([string] $functionAppName, [string] $resourceGroup, [string] $npmpackage)

$creds = Invoke-AzureRmResourceAction -ResourceGroupName $resourceGroup -ResourceType Microsoft.Web/sites/config `
            -ResourceName $functionAppName/publishingcredentials -Action list -ApiVersion 2015-08-01 -Force

$username = $creds.Properties.PublishingUserName
$password = $creds.Properties.PublishingPassword


$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password)))

$userAgent = "powershell/1.0"
$apiUrl = "https://$functionAppName.scm.azurewebsites.net/api/command"
$command = '{"command": "npm install ' + $npmpackage + '","dir": "site\\wwwroot"}'

Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method POST -Body $command -ContentType "application/json"

Now all we have left is to execute this script in a Azure Powershell Task that will install the npm pacakge during the release, image bellow shows the Release setup and as you can see the parameters are added in the “Script Arguments” input area. I’ve also added the zcript to a shared repo andlinked the build setup to be able to share the script and manage it in one place.

If you are using package.json files we can make sure all packages are installed via npm install command, let’s see how the following could look like with a package.json file:

Modifying the PowerShell script will then give us the following:

functionAppName: the name of the Function App
resourceGroupName: the name of the resource group that contains the Function App
functionfolder: the name of the folder that the function is in (same name as the function)

param([string] $functionAppName, [string] $resourceGroup, [string] $functionfolder)

$creds = Invoke-AzureRmResourceAction -ResourceGroupName $resourceGroup -ResourceType Microsoft.Web/sites/config `
            -ResourceName $functionAppName/publishingcredentials -Action list -ApiVersion 2015-08-01 -Force

$username = $creds.Properties.PublishingUserName
$password = $creds.Properties.PublishingPassword

$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password)))

$userAgent = "powershell/1.0"
$apiUrl = "https://$functionAppName.scm.azurewebsites.net/api/command"
$command = '{"command": "npm install","dir": "site\\wwwroot\\'+ $functionfolder +'"}'

Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method POST -Body $command -ContentType "application/json"

Summary:

I like to automate these tasks since it will give me a “easier” Release and a more reliable Release, but as for now it’s hard to verify that the package is installed and if it’s installed previously so a verification step that the function is loaded correctly is something that might be needed.

Recomended is to use the package.json approach since it shows the dependencies and allow to easyily add new packages, but jeep in mind that it also gives some delays when running the npm install command since it will install alot of packages the first time.

Azure Functions - Manage Application Settings via ARM

2017-10-23T00:00:00+00:00

One of the most common task when working with deployments is the need to handle is application settings. This is where we are adding environment variables to the current Azure Function App, the environment variables often has diffrent values in diffrent environmnets (Dev/Test/Prod). So we need to manage them to make sure that they have the correct values.

These can be managed manually since they are rarely changed but it requires dicipline and good release documentation and sometimes quite advanced procedures to get the information, some of these cases are i.e. getting the URL of a Logic App, a value from Key Vault or we might just walue quality and reliable deployments.

I prefer to manage these settings via ARM templates since it gives me the possiblity to get values from Key Vault or automate the process to get values from other Azure resources and at the same time gives me control and robustness so I know that the settings are set and that the values are correct. A note to this, make sure that operations persons are aware that settings are been set from ARM template deployment since if they do changes directly in the Azure functions Application Settings tab they will be overwriten at the next deployment and that might cause problems.

Let’s look in to the ARM template, I’ve just copied this from the Azure Portal (automation option when creating a new Azure Function).

{
  "parameters": {
    "name": {
      "type": "string"
    },
    "storageName": {
      "type": "string"
    },
    "location": {
      "type": "string"
    }
  },
  "resources": [
    {
      "name": "[parameters('name')]",
      "type": "Microsoft.Web/sites",
      "dependsOn": [
        "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageName'))]",
        "[resourceId('microsoft.insights/components/', parameters('name'))]"
      ],
      "properties": {
        "siteConfig": {
          "appSettings": [
            {
              "name": "AzureWebJobsDashboard",
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2015-05-01-preview').key1)]"
            },
            {
              "name": "AzureWebJobsStorage",
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2015-05-01-preview').key1)]"
            },
            {
              "name": "FUNCTIONS_EXTENSION_VERSION",
              "value": "~1"
            },
            {
              "name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2015-05-01-preview').key1)]"
            },
            {
              "name": "WEBSITE_CONTENTSHARE",
              "value": "[concat(toLower(parameters('name')), 'a217')]"
            },
            {
              "name": "WEBSITE_NODE_DEFAULT_VERSION",
              "value": "6.5.0"
            },
            {
              "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
              "value": "[reference(resourceId('microsoft.insights/components/', parameters('name')), '2015-05-01').InstrumentationKey]"
            }
          ]
        },
        "name": "[parameters('name')]",
        "clientAffinityEnabled": false
      },
      "apiVersion": "2016-03-01",
      "location": "[parameters('location')]",
      "kind": "functionapp"
    },
    {
      "apiVersion": "2015-05-01-preview",
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[parameters('storageName')]",
      "location": "[parameters('location')]",
      "properties": {
        "accountType": "Standard_LRS"
      }
    },
    {
      "apiVersion": "2015-05-01",
      "name": "[parameters('name')]",
      "type": "microsoft.insights/components",
      "location": "West Europe",
      "tags": {
        "[concat('hidden-link:', resourceGroup().id, '/providers/Microsoft.Web/sites/', parameters('name'))]": "Resource"
      },
      "properties": {
        "ApplicationId": "[parameters('name')]",
        "Request_Source": "IbizaWebAppExtensionCreate"
      }
    }
  ],
  "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0"
}

The interesting section is the appSettings under the siteConfig shown bellow.

Here is the standard properties that are used when deploying a Azure Function and here we can add new Application Settings. (note that ARM functions are used to get keys to the storage account)

"properties": {
        "siteConfig": {
          "appSettings": [
            {
              "name": "AzureWebJobsDashboard",
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2015-05-01-preview').key1)]"
            },
            {
              "name": "AzureWebJobsStorage",
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2015-05-01-preview').key1)]"
            },
            {
              "name": "FUNCTIONS_EXTENSION_VERSION",
              "value": "~1"
            },
            {
              "name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
              "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2015-05-01-preview').key1)]"
            },
            {
              "name": "WEBSITE_CONTENTSHARE",
              "value": "[concat(toLower(parameters('name')), 'a217')]"
            },
            {
              "name": "WEBSITE_NODE_DEFAULT_VERSION",
              "value": "6.5.0"
            },
            {
              "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
              "value": "[reference(resourceId('microsoft.insights/components/', parameters('name')), '2015-05-01').InstrumentationKey]"
            }
          ]
        },

Adding a new property to the list is easy and adding a new key customkey looks like this, and we also add a new ARM parameter so we can change the value between environments

    {
      "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
      "value": "[reference(resourceId('microsoft.insights/components/', parameters('name')), '2015-05-01').InstrumentationKey]"
    },
    {
      "name": "customkey",
      "value": "[parameters('customkeyparam')]"
    }
  ]
},

Added parameter:

"location": {
  "type": "string"
},
"customkeyparam" :{
  "type": "string"
}

And now we can use this with a parameter file, I copied the paramter file the same way and now we can update this parameter file and adding the new parameter:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "name": {
      "value": "ibizmalotest"
    },
    "storageName": {
      "value": "ibizmalotest912c"
    },
    "location": {
      "value": "West Europe"
    }
    "customkeyparam" :{
      "value": "MyCustomValueSetViaARM"
    }
  }
}

After doing a Resource Group Deployment, we can go in and see the new setting in the Application Setting on the Azure Function App and start using it.

Summary:

I like this approach since we can make sure that parameters are set and prepare before deployment without needing to change values in the Function App, I also love the ARM functions that we can use during deployments to automate processes or just get values from Azure Key Vaults to make it easier to manage secrets (note that secrets are in plain text in application settings for the users who has access).

The ARM functions can really speed up and automate the deployment processes even further, here we can as shown aboive get keys to storage account or get the URL of a Logic App during deployment time, wich is a complex/time consuming task and when using ARM functions there is also a garantee that the Logic App is deployed and ready to be used.

Creating a good ARM template will make it easier to move between environments and it will add quality to your deployments.

But remember to make sure that changes that are made directly to the Function App Application Settings are reflected to the parameter files and arm templates aswell to prevent changes to be overwriten.

Sample getting the URL of a Locic App (assumes added parameters for Resource Group and Logic App name and the trigger name). (template file)

    {
      "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
      "value": "[reference(resourceId('microsoft.insights/components/', parameters('name')), '2015-05-01').InstrumentationKey]"
    },
    {
      "name": "LogicAppUrl",
      "value": "[listCallbackUrl(resourceId(parameters('logicApp_resourcegroup'),'Microsoft.Logic/workflows/triggers', parameters('logicApp_name'),parameters('logicApp_trigger')), providers('Microsoft.Logic', 'workflows').apiVersions[0]).value]"
    }
  ]
},

Sample Azure Key Vault get a specific Value from Key Vault. (parameter file)

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "name": {
      "value": "ibizmalotest"
    },
    "storageName": {
      "value": "ibizmalotest912c"
    },
    "location": {
      "value": "West Europe"
    }
    "customkeyparam" :{
      "reference": {
        "keyVault": {
          "id": "/subscriptions/fake-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/testenvironment/providers/Microsoft.KeyVault/vaults/ibizmalotest"
        },
        "secretName": "customkeysecretname"
      }
    }
  }
}

API Management ARM Template Creator

2017-09-11T00:00:00+00:00

ARM support was release in the July rlease of API Management (read more here) and this is relly a great improvement on the deployment experience of API Management. For samples on ARM templates released by the product group got the Sample Github page

But there is some “blocking” things here, we are not using ARM templates when devloping, so how do we get the ARM Templates? First you can download the entire API Management instance ARM specification directly in the portal, Automation Script

But when starting to look into this I got a little dissapointed, it’s to mutch hardcoded values like backend URL’s parameters and there is for me unwanted things aswell as Users, Groups, Subscriptions etc. All that could be sorted out and scripted (alot of work but could be done) but the real deal breaker using the automation script was the missing request/response representation and the missing queryParameters wich lead to crahsing imports.

It’s import to understand the Environment reference and boundries, we don’t want cross reference communcation that is not intended:

When doing a deployment we want as little manual work as possible and above we have 2 issues, manual configuration equals downtime and sample messagas is one of the best things when onboarding new API consumers we can’t ignore this.

Let’s look at an example, this is how an operation is looking (note the sample):

And when this api is exported via the Automation Script the sample is not part of the object returned:

 {
    "comments": "Generalized from resource: '/subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/apis/58a3138ca9f54041296baffd/operations/58a313e40647c00eecf8d408'.",
    "type": "Microsoft.ApiManagement/service/apis/operations",
    "name": "[concat(parameters('service_ibizmalo_name'), '/', parameters('apis_58a3138ca9f54041296baffd_name'), '/', parameters('operations_58a313e40647c00eecf8d408_name'))]",
    "apiVersion": "2017-03-01",
    "scale": null,
    "properties": {
        "displayName": "Conversion Rate",
        "method": "POST",
        "urlTemplate": "/conversionrate",
        "description": "<br><b>Get conversion rate from one currency to another currency <b><br><p><b><font color='#000080' size='1' face='Verdana'><u>Differenct currency Code and Names around the world</u></font></b></p><blockquote><p><font face='Verdana' size='1'>AFA-Afghanistan Afghani<br>ALL-Albanian Lek<br>DZD-Algerian Dinar<br>ARS-Argentine Peso<br>AWG-Aruba Florin<br>AUD-Australian Dollar<br>BSD-Bahamian Dollar<br>BHD-Bahraini Dinar<br>BDT-Bangladesh Taka<br>BBD-Barbados Dollar<br>BZD-Belize Dollar<br>BMD-Bermuda Dollar<br>BTN-Bhutan Ngultrum<br>BOB-Bolivian Boliviano<br>BWP-Botswana Pula<br>BRL-Brazilian Real<br>GBP-British Pound<br>BND-Brunei Dollar<br>BIF-Burundi Franc<br>XOF-CFA Franc (BCEAO)<br>XAF-CFA Franc (BEAC)<br>KHR-Cambodia Riel<br>CAD-Canadian Dollar<br>CVE-Cape Verde Escudo<br>KYD-Cayman Islands Dollar<br>CLP-Chilean Peso<br>CNY-Chinese Yuan<br>COP-Colombian Peso<br>KMF-Comoros Franc<br>CRC-Costa Rica Colon<br>HRK-Croatian Kuna<br>CUP-Cuban Peso<br>CYP-Cyprus Pound<br>CZK-Czech Koruna<br>DKK-Danish Krone<br>DJF-Dijibouti Franc<br>DOP-Dominican Peso<br>XCD-East Caribbean Dollar<br>EGP-Egyptian Pound<br>SVC-El Salvador Colon<br>EEK-Estonian Kroon<br>ETB-Ethiopian Birr<br>EUR-Euro<br>FKP-Falkland Islands Pound<br>GMD-Gambian Dalasi<br>GHC-Ghanian Cedi<br>GIP-Gibraltar Pound<br>XAU-Gold Ounces<br>GTQ-Guatemala Quetzal<br>GNF-Guinea Franc<br>GYD-Guyana Dollar<br>HTG-Haiti Gourde<br>HNL-Honduras Lempira<br>HKD-Hong Kong Dollar<br>HUF-Hungarian Forint<br>ISK-Iceland Krona<br>INR-Indian Rupee<br>IDR-Indonesian Rupiah<br>IQD-Iraqi Dinar<br>ILS-Israeli Shekel<br>JMD-Jamaican Dollar<br>JPY-Japanese Yen<br>JOD-Jordanian Dinar<br>KZT-Kazakhstan Tenge<br>KES-Kenyan Shilling<br>KRW-Korean Won<br>KWD-Kuwaiti Dinar<br>LAK-Lao Kip<br>LVL-Latvian Lat<br>LBP-Lebanese Pound<br>LSL-Lesotho Loti<br>LRD-Liberian Dollar<br>LYD-Libyan Dinar<br>LTL-Lithuanian Lita<br>MOP-Macau Pataca<br>MKD-Macedonian Denar<br>MGF-Malagasy Franc<br>MWK-Malawi Kwacha<br>MYR-Malaysian Ringgit<br>MVR-Maldives Rufiyaa<br>MTL-Maltese Lira<br>MRO-Mauritania Ougulya<br>MUR-Mauritius Rupee<br>MXN-Mexican Peso<br>MDL-Moldovan Leu<br>MNT-Mongolian Tugrik<br>MAD-Moroccan Dirham<br>MZM-Mozambique Metical<br>MMK-Myanmar Kyat<br>NAD-Namibian Dollar<br>NPR-Nepalese Rupee<br>ANG-Neth Antilles Guilder<br>NZD-New Zealand Dollar<br>NIO-Nicaragua Cordoba<br>NGN-Nigerian Naira<br>KPW-North Korean Won<br>NOK-Norwegian Krone<br>OMR-Omani Rial<br>XPF-Pacific Franc<br>PKR-Pakistani Rupee<br>XPD-Palladium Ounces<br>PAB-Panama Balboa<br>PGK-Papua New Guinea Kina<br>PYG-Paraguayan Guarani<br>PEN-Peruvian Nuevo Sol<br>PHP-Philippine Peso<br>XPT-Platinum Ounces<br>PLN-Polish Zloty<br>QAR-Qatar Rial<br>ROL-Romanian Leu<br>RUB-Russian Rouble<br>WST-Samoa Tala<br>STD-Sao Tome Dobra<br>SAR-Saudi Arabian Riyal<br>SCR-Seychelles Rupee<br>SLL-Sierra Leone Leone<br>XAG-Silver Ounces<br>SGD-Singapore Dollar<br>SKK-Slovak Koruna<br>SIT-Slovenian Tolar<br>SBD-Solomon Islands Dollar<br>SOS-Somali Shilling<br>ZAR-South African Rand<br>LKR-Sri Lanka Rupee<br>SHP-St Helena Pound<br>SDD-Sudanese Dinar<br>SRG-Surinam Guilder<br>SZL-Swaziland Lilageni<br>SEK-Swedish Krona<br>TRY-Turkey Lira<br>CHF-Swiss Franc<br>SYP-Syrian Pound<br>TWD-Taiwan Dollar<br>TZS-Tanzanian Shilling<br>THB-Thai Baht<br>TOP-Tonga Pa'anga<br>TTD-Trinidad&amp;amp;Tobago Dollar<br>TND-Tunisian Dinar<br>TRL-Turkish Lira<br>USD-U.S. Dollar<br>AED-UAE Dirham<br>UGX-Ugandan Shilling<br>UAH-Ukraine Hryvnia<br>UYU-Uruguayan New Peso<br>VUV-Vanuatu Vatu<br>VEB-Venezuelan Bolivar<br>VND-Vietnam Dong<br>YER-Yemen Riyal<br>YUM-Yugoslav Dinar<br>ZMK-Zambian Kwacha<br>ZWD-Zimbabwe Dollar</font></p></blockquote>",
        "policies": null
    },
    "dependsOn": [
        "[resourceId('Microsoft.ApiManagement/service', parameters('service_ibizmalo_name'))]",
        "[resourceId('Microsoft.ApiManagement/service/apis', parameters('service_ibizmalo_name'), parameters('apis_58a3138ca9f54041296baffd_name'))]"
    ]
},

To be compared with the representation recieved when extracting the API using the REST interface:

{
  {
    "id": "/subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/apis/58a3138ca9f54041296baffd/operations/58a313e40647c00eecf8d408",
    "type": "Microsoft.ApiManagement/service/apis/operations",
    "name": "58a313e40647c00eecf8d408",
    "properties": {
      "displayName": "Conversion Rate",
      "method": "POST",
      "urlTemplate": "/conversionrate",
      "templateParameters": [],
      "description": "<br><b>Get conversion rate from one currency to another currency <b><br><p><b><font color='#000080' size='1' face='Verdana'><u>Differenct currency Code and Names around the world</u></font></b></p><blockquote><p><font face='Verdana' size='1'>AFA-Afghanistan Afghani<br>ALL-Albanian Lek<br>DZD-Algerian Dinar<br>ARS-Argentine Peso<br>AWG-Aruba Florin<br>AUD-Australian Dollar<br>BSD-Bahamian Dollar<br>BHD-Bahraini Dinar<br>BDT-Bangladesh Taka<br>BBD-Barbados Dollar<br>BZD-Belize Dollar<br>BMD-Bermuda Dollar<br>BTN-Bhutan Ngultrum<br>BOB-Bolivian Boliviano<br>BWP-Botswana Pula<br>BRL-Brazilian Real<br>GBP-British Pound<br>BND-Brunei Dollar<br>BIF-Burundi Franc<br>XOF-CFA Franc (BCEAO)<br>XAF-CFA Franc (BEAC)<br>KHR-Cambodia Riel<br>CAD-Canadian Dollar<br>CVE-Cape Verde Escudo<br>KYD-Cayman Islands Dollar<br>CLP-Chilean Peso<br>CNY-Chinese Yuan<br>COP-Colombian Peso<br>KMF-Comoros Franc<br>CRC-Costa Rica Colon<br>HRK-Croatian Kuna<br>CUP-Cuban Peso<br>CYP-Cyprus Pound<br>CZK-Czech Koruna<br>DKK-Danish Krone<br>DJF-Dijibouti Franc<br>DOP-Dominican Peso<br>XCD-East Caribbean Dollar<br>EGP-Egyptian Pound<br>SVC-El Salvador Colon<br>EEK-Estonian Kroon<br>ETB-Ethiopian Birr<br>EUR-Euro<br>FKP-Falkland Islands Pound<br>GMD-Gambian Dalasi<br>GHC-Ghanian Cedi<br>GIP-Gibraltar Pound<br>XAU-Gold Ounces<br>GTQ-Guatemala Quetzal<br>GNF-Guinea Franc<br>GYD-Guyana Dollar<br>HTG-Haiti Gourde<br>HNL-Honduras Lempira<br>HKD-Hong Kong Dollar<br>HUF-Hungarian Forint<br>ISK-Iceland Krona<br>INR-Indian Rupee<br>IDR-Indonesian Rupiah<br>IQD-Iraqi Dinar<br>ILS-Israeli Shekel<br>JMD-Jamaican Dollar<br>JPY-Japanese Yen<br>JOD-Jordanian Dinar<br>KZT-Kazakhstan Tenge<br>KES-Kenyan Shilling<br>KRW-Korean Won<br>KWD-Kuwaiti Dinar<br>LAK-Lao Kip<br>LVL-Latvian Lat<br>LBP-Lebanese Pound<br>LSL-Lesotho Loti<br>LRD-Liberian Dollar<br>LYD-Libyan Dinar<br>LTL-Lithuanian Lita<br>MOP-Macau Pataca<br>MKD-Macedonian Denar<br>MGF-Malagasy Franc<br>MWK-Malawi Kwacha<br>MYR-Malaysian Ringgit<br>MVR-Maldives Rufiyaa<br>MTL-Maltese Lira<br>MRO-Mauritania Ougulya<br>MUR-Mauritius Rupee<br>MXN-Mexican Peso<br>MDL-Moldovan Leu<br>MNT-Mongolian Tugrik<br>MAD-Moroccan Dirham<br>MZM-Mozambique Metical<br>MMK-Myanmar Kyat<br>NAD-Namibian Dollar<br>NPR-Nepalese Rupee<br>ANG-Neth Antilles Guilder<br>NZD-New Zealand Dollar<br>NIO-Nicaragua Cordoba<br>NGN-Nigerian Naira<br>KPW-North Korean Won<br>NOK-Norwegian Krone<br>OMR-Omani Rial<br>XPF-Pacific Franc<br>PKR-Pakistani Rupee<br>XPD-Palladium Ounces<br>PAB-Panama Balboa<br>PGK-Papua New Guinea Kina<br>PYG-Paraguayan Guarani<br>PEN-Peruvian Nuevo Sol<br>PHP-Philippine Peso<br>XPT-Platinum Ounces<br>PLN-Polish Zloty<br>QAR-Qatar Rial<br>ROL-Romanian Leu<br>RUB-Russian Rouble<br>WST-Samoa Tala<br>STD-Sao Tome Dobra<br>SAR-Saudi Arabian Riyal<br>SCR-Seychelles Rupee<br>SLL-Sierra Leone Leone<br>XAG-Silver Ounces<br>SGD-Singapore Dollar<br>SKK-Slovak Koruna<br>SIT-Slovenian Tolar<br>SBD-Solomon Islands Dollar<br>SOS-Somali Shilling<br>ZAR-South African Rand<br>LKR-Sri Lanka Rupee<br>SHP-St Helena Pound<br>SDD-Sudanese Dinar<br>SRG-Surinam Guilder<br>SZL-Swaziland Lilageni<br>SEK-Swedish Krona<br>TRY-Turkey Lira<br>CHF-Swiss Franc<br>SYP-Syrian Pound<br>TWD-Taiwan Dollar<br>TZS-Tanzanian Shilling<br>THB-Thai Baht<br>TOP-Tonga Pa'anga<br>TTD-Trinidad&amp;amp;Tobago Dollar<br>TND-Tunisian Dinar<br>TRL-Turkish Lira<br>USD-U.S. Dollar<br>AED-UAE Dirham<br>UGX-Ugandan Shilling<br>UAH-Ukraine Hryvnia<br>UYU-Uruguayan New Peso<br>VUV-Vanuatu Vatu<br>VEB-Venezuelan Bolivar<br>VND-Vietnam Dong<br>YER-Yemen Riyal<br>YUM-Yugoslav Dinar<br>ZMK-Zambian Kwacha<br>ZWD-Zimbabwe Dollar</font></p></blockquote>",
      "request": {
        "description": "ConversionRequest",
        "queryParameters": [],
        "headers": [],
        "representations": [
          {
            "contentType": "application/json",
            "sample": "{     \"fromCurrency\": \"SEK\",     \"toCurrency\": \"USD\" }",
            "schemaId": "322649e9-bfd5-4c93-8ad1-3fc1780e8045",
            "typeName": "ConversionRate"
          }
        ]
      },
      "responses": [
        {
          "statusCode": 200,
          "description": "ConversionResult",
          "representations": [
            {
              "contentType": "application/json",
              "sample": "{\r\n    \"rate\": 1.0\r\n}",
              "schemaId": "322649e9-bfd5-4c93-8ad1-3fc1780e8045",
              "typeName": "ConversionRateResponse"
            }
          ],
          "headers": []
        }
      ],
      "policies": null
    }
  }
}

We can clearly see that the REST object contains far more inforamtion and since we are working withe environment setups with DEV/TEST/PROD it would be dissapointing to not be able to provide samples in DEV but not in TEST and PRODUCTION (where it makes most sense):

So this was the start of my new project API Management ARM Template Creator

As some of you know I’ve been contributing alot to the Logic App Template Creator: https://github.com/jeffhollan/LogicAppTemplateCreator so I thought I’ll do something similar for API Management.

While starting this I added a few more requirements

Possible to only extract one specific api, but get all required linked attributes such as Named Values.
Parameterized URL’s (both API and backend)
Parameterized Named Values value
Automated url/Named values handling for Logic Apps
Parameterized values on all changeable values that is needed for groups, products, authorization servers etc

The first release is made today and the project can be found at my GIT site: https://github.com/MLogdberg/APIManagementARMTemplateCreator

How does it work?

Well it’s no magic, I’ve just combined the result from the REST API’s to ARM Templates and along the way added some ARM functionality to improve the development experience.

Let’s show with a sample with one of my API’s: I have a “simple” rest API, that has two GET operations, with a policy for api and one for each operation, each operation need’s an authentication key that is stored in Named Values.

The key components that need’s to be changed are the URL and the authentication key used in the request to the backend.

The URL

The authenticationkey in the policy ()

The authenticationkey in Named Values ()

Even for a rather simple API there are som places that need’s to be changed and in order to prevent downtime on your API this is needed to be changed before/during deployment. So using the PowerShell module I can extract a ARM template that will represent this API.

Let’s look at some samples:

Manually created REST API

Parameterization of URL, the API looks like this and as you can see the URL is parameterized with [parameters(‘order_serviceUrl’)]

{
      "comments": "Generated for resource /subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/apis/order",
      "type": "Microsoft.ApiManagement/service/apis",
      "name": "[concat(parameters('service_ibizmalo_name'), '/' ,parameters('api_order_name'))]",
      "apiVersion": "2017-03-01",
      "properties": {
        "displayName": "Order",
        "apiRevision": "[parameters('order_apiRevision')]",
        "description": "Azure Logic App.",
        "serviceUrl": "[parameters('order_serviceUrl')]",
        "path": "api/v1/order",
        "protocols": [
          "https"
        ],

Parameterization of authentication key, the Named Value (Properties it’s called in ARM) also has it’s value parameterized with [parameters(‘trafficauthkey_value’)]

{
      "comments": "Generated for resource /subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/properties/trafficauthkey",
      "type": "Microsoft.ApiManagement/service/properties",
      "name": "[concat(parameters('service_ibizmalo_name'), '/' ,parameters('property_trafficauthkey_name'))]",
      "apiVersion": "2017-03-01",
      "properties": {
        "displayName": "trafficauthkey",
        "value": "[parameters('trafficauthkey_value')]",
        "tags": null,
        "secret": true
      },
      "resources": [],
      "dependsOn": []
    }

And due to these parameters it’s easy to change the values between environments and deliver robust and quality deployments.

Logic App

When generating the API from a Logic App the needed changes between enviroments are to change the following things: the resource group, the name of the logic app and the trigger name. The rest will be automated according the standard generation in API Management for Logic Apps.

Description of the standard generation of policy for Logic App instances, first there is a rewrite URL that will change the URL to match the Logic App and also add the sv and sig values (authentication) in a Named Value (Property) in to the URL. Second step is to set the Base URL of the Call to match the Logic App URL.

Lastly there is a comment section added with the path to the original Azure Resource (our Logic App) used when generating the policy. This is something we are using to automate the URL and authentication values.

The policy will be updated so that the policyContent will use the listCallbackUrl ARM function to retrieve the Logic App’s base url.

 {
  "comments": "Generated for resource /subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/apis/order/operations/59a6b4730d691205f068a8be/policies/policy",
  "type": "Microsoft.ApiManagement/service/apis/operations/policies",
  "name": "[concat(parameters('service_ibizmalo_name'), '/' ,parameters('api_order_name'), '/' ,parameters('operations_59a6b4730d691205f068a8be_name'), '/' ,parameters('policy_policy_name'))]",
  "apiVersion": "2017-03-01",
  "properties": {
    "policyContent": "[Concat('<policies>\r\n  <inbound>\r\n    <rewrite-uri id=\"apim-generated-policy\" template=\"?api-version=2016-06-01&amp;sp=/triggers/request/run&amp;\" />\r\n    <set-backend-service id=\"apim-generated-policy\" base-url=\"',listCallbackUrl(resourceId(parameters('logicApp_INT001-GetOrderInfo_resourcegroup'),'Microsoft.Logic/workflows/triggers', parameters('logicApp_INT001-GetOrderInfo_name'),parameters('logicApp_INT001-GetOrderInfo_trigger')), providers('Microsoft.Logic', 'workflows').apiVersions[0]).basePath,'\" />\r\n    <base />\r\n    <set-header name=\"Ocp-Apim-Subscription-Key\" exists-action=\"delete\" />\r\n  </inbound>\r\n  <outbound>\r\n    <base />\r\n  </outbound>\r\n  <backend>\r\n    <base />\r\n    <!-- { \"azureResource\": { \"type\": \"logicapp\", \"id\": \"/subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.Logic/workflows/INT001-GetOrderInfo/triggers/request\" } } -->\r\n  </backend>\r\n</policies>')]"
  },
  "resources": [],
  "dependsOn": [
    "[resourceId('Microsoft.ApiManagement/service/apis', parameters('service_ibizmalo_name') ,parameters('api_order_name'))]",
    "[resourceId('Microsoft.ApiManagement/service/apis/operations', parameters('service_ibizmalo_name'), parameters('api_order_name'), parameters('operations_59a6b4730d691205f068a8be_name'))]"
  ]
}

The Named Value (Property) will be automated the same way, using the listCallbackUrl ARM function.

{
  "comments": "Generated for resource /subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/properties/59a6b478285385ab3dfbb752",
  "type": "Microsoft.ApiManagement/service/properties",
  "name": "[concat(parameters('service_ibizmalo_name'), '/' ,parameters('property_59a6b478285385ab3dfbb752_name'))]",
  "apiVersion": "2017-03-01",
  "properties": {
    "displayName": "orderrequest59a6b4783fb21a7984df42ae",
    "value": "[concat('sv=',listCallbackUrl(resourceId(parameters('logicApp_INT001-GetOrderInfo_resourcegroup'),'Microsoft.Logic/workflows/triggers', parameters('logicApp_INT001-GetOrderInfo_name'),parameters('logicApp_INT001-GetOrderInfo_trigger')), providers('Microsoft.Logic', 'workflows').apiVersions[0]).queries.sv,'&sig=',listCallbackUrl(resourceId(parameters('logicApp_INT001-GetOrderInfo_resourcegroup'),'Microsoft.Logic/workflows/triggers', parameters('logicApp_INT001-GetOrderInfo_name'),parameters('logicApp_INT001-GetOrderInfo_trigger')), providers('Microsoft.Logic', 'workflows').apiVersions[0]).queries.sig)]",
    "tags": [],
    "secret": true
  },
  "resources": [],
  "dependsOn": []
}

By using the listCallbackUrl function we can retrieve the values from the specific Logic App at deployment time, removing all manual handling of URL’s and authentication values when working with API’s exposing Logic Apps.

SOAP

So when SOAP was introduced to API Management a new resource was added, Backend this is unfortunally not editable in the GUI yet but we can do changes on with ARM deployments, the REST and PS API’s. So in order to create a full ARM template for SOAP we need that extra resource and all other allready described information.

In my sample API Manegement instance my SOAP Currency Converter API is generated with a backend that has a parameter [parameters(‘f9211da9-8ce1-48c7-8e13-e38bc5dbcbcd_url’)] to be used to change the URL.

{
  "comments": "Generated for resource /subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/PreDemoTest/providers/Microsoft.ApiManagement/service/ibizmalo/backends/f9211da9-8ce1-48c7-8e13-e38bc5dbcbcd",
  "type": "Microsoft.ApiManagement/service/backends",
  "name": "[concat(parameters('service_ibizmalo_name'), '/' ,parameters('backend_f9211da9-8ce1-48c7-8e13-e38bc5dbcbcd_name'))]",
  "apiVersion": "2017-03-01",
  "properties": {
    "title": "CurrencyConverter Soap Backend",
    "description": "CurrencyConverter Soap Backend",
    "url": "[parameters('f9211da9-8ce1-48c7-8e13-e38bc5dbcbcd_url')]",
    "protocol": "soap",
    "properties": {},
    "tls": {}
  },
  "resources": []
},

Get Started

So how do you get started? First of download the repo https://github.com/MLogdberg/APIManagementARMTemplateCreator, compile it and use this PowerShell script (replace the values with appropiate ones from your subscription and API Management instance).

#if you have problem with execution policy execute this in a administrator runned powershell window.
#Set-ExecutionPolicy -ExecutionPolicy Unrestricted

Import-Module "C:\temp\APIManagementARMTemplateCreator\APIManagementTemplate\bin\Debug\APIManagementTemplate.dll"


#Set the name of the API Mangement instance
$apimanagementname = 'ibizmalo'

#Set the resource group 
$resourcegroupname = 'PreDemoTest' # 'app-tst-cs-int-Migrated'
#Set the subscription id 
$subscriptionid = 'c107df29-a4af-4bc9-a733-f88f0eaa4296'#'d4baa1e9-15f5-4c85-bb3e-1e108dc79b00'#
#Set the tenant to use when login ing, make sure it has the right tennant
$tenant = 'mattiaslogdbergibizsolution.onmicrosoft.com'

#optional set filter for a specific api (using standard REST filter, with path we can select api based on the API path)
#$filter = "path eq 'api/v1/order'"
#$filter = "path eq 'api/v1/currencyconverter'"

 
#setting the output filename
$filenname = $apimanagementname + '.json'

Get-APIManagementTemplate -APIFilters $filter -APIManagement $apimanagementname -ResourceGroup $resourcegroupname -SubscriptionId $subscriptionid -TenantName $tenant -ExportPIManagementInstance $false  | Out-File $filenname

Upcoming

More functions and better support for authentication is on the todo list, also smarter/simpler product handling, here are my bullet points:

Automated support for all Azure Resources.
Operations Schema support.
Retrieve authroization servers linked to API, and parameterize them as needed.
Better support for authorization servers.
Support for Products and linking API’s to products

API Management concurrency control

2017-09-07T00:00:00+00:00

As I was mentioning in the last post with concurrency control for Logic App, it’s important in the distributed world today to make sure to not flood our backend services since there are more restrictions in the shared and distributed landscape than it has been in the good old days OnPrem where we had full control over all our services. That means that we need to have control over our integration solutions so they don’t spin away like crazy and causes unwanted flooding to our backend systems.

Historically we could control the request towards our backends system with Rate-Call-Limit and Qutoas to make sure that a maximum of request was made by a consumers during a time period, that could be minutes,hours,days,months.

But in the latest release the API Management the team added amongst other concurrency control. This means that we now also can control/restrict the number of concurrent calls to our backend service.

I.e. we have a SaaS service that allows 20 concurrent calls, all calls without that limit will be declined. This was hard to control from API Management and all the responsibility was left to the consumers. We had a case where the consumer push 1000 calls in a few miliseconds, distributed world is lovely! But the backend service couldnt handle it and most of the failed and a new batch was fired and yes the story repeated itself. It ended with the consumer needing to change their solution (yes I agree that was the best way) but we had no chance off protecting the backend from the flood except Rate-Limit but that was not an optimal solution.

Now whit Concurrency control we can garantée that the backend system don’t have more than 20 concurrent calls and all calls after are queued up to a maximum limit, and processed right away as one of the 20 concurrent calls are finished. Making sure to not waste resoruces in retrys and idle time. Using the Concurrency control is simple here is the policy: read more here

<limit-concurrency key="expression" max-count="number" timeout="in seconds" max-queue-length="number">
        <!— nested policy statements -->  
</limit-concurrency>

So let’s show a sample, I’ve created a Logic App to act as a backend service, it hasen’t mutch logic but a delay to simulate really heavy work (20 seconds)

Part 1: A simple flow will be a simple Logic app to Logic App with no restrictions:

And now let’s see how this looks like when consumed by another Logic App in a foreach loop (without any concurrency restrictions).

As assumed the execution takes abit longer than 20 seconds:

Part 2: Adding concurrency control by exposing the Backend Logic App via API Management

The Logic App is imported to API Management and the policy Concurrency Control is added:

And the policy looks like:

<policies>
	<inbound>
		<base />
		<rewrite-uri id="apim-generated-policy" template="?api-version=2016-06-01&amp;sp=/triggers/request/run&amp;" />
		<set-backend-service id="apim-generated-policy" base-url="https://prod-40.westeurope.logic.azure.com/workflows/24b778e6805344e686536a203ee47bce/triggers/request/paths/invoke" />
		<set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />
	</inbound>
	<backend>
		<limit-concurrency key="constantstring" max-count="2" timeout="120" max-queue-length="100">
			<forward-request timeout="120" />
		</limit-concurrency>
		<!--{ "azureResource": { "type": "logicapp", "id": "/subscriptions/c107df29-a4af-4bc9-a733-f88f0eaa4296/resourceGroups/Concurrency/providers/Microsoft.Logic/workflows/INT0002-SingelInstance/triggers/request" } }-->
	</backend>
	<outbound>
		<base />
	</outbound>
	<on-error>
		<base />
	</on-error>
</policies>

To demonstrat the functionallity a new Logic App is now used , same logic as before but using the API Management to access the backend Logic App.

Executing the Logic App will now take 4 minutes! since the concurrency is set to 2.

So that is great now we have prevented to many concurrent calls to our backend system, but is it all good? Let’s look in to the log of our “backend system”

We can see that there are 2 failed runs and 22 in total runs, but as you remember the list had 20 records so we only wanted 20 request to our “backend system”.

The failed ones has failed due to timeout and that will also make the Logic App executing a retry, ending with a total of 22 calls to the backend.

Summary: This is truly a nice feature that we will use alot but we also need to understand the nature of the API and how to configure it correctly to prevent unwanted behavior. In a flow where resending the information is not allowed, the queue lenght should be set as low as possible to prevent the retry behavior of clients resending the information even if it’s not wanted. Or we might want to use other techniques for that kind of flows.

All in all a great feature and wery usefull one, but as allways make sure to understand the feature and the requirements of the protected API when configuring it.

Logic Apps concurrency control

2017-08-29T00:00:00+00:00

In a distributed world with increasing amount of moving parts we need to make sure to not flood our services since there are more restrictions in the shared and distributed landscape than it has been in the good old days where we had full control over all our services. That means that we need to have control over our solutions so they don’t spin away like crazy.

On this topic Logic Apps has earlier had problem, when looping over an array in a foreach loop and calling a connector we could choose between 20 concurrent actions or sequentially call it one by one. This mean that if 20 parallel actions was flooding the destination we had to make it to sequantial call the destination and therefore also taking alot longer. When working with batches this is not always possible since it might take to long time, so sometimes we had to create two or more parallel flows in Logic Apps to make suer that we could keep the time limit.

Making the flow complex and look “uggly”:

Newly released concurrency control!

But fortunately there is now a newly released object property on the for each action that will help us, only available in CodeView for now but that will work just fine.

The new property looks like:

"runtimeConfiguration": {
    "concurrency": {
        "repetitions": 2
    }
},

And setting the repetitions to 2 will be the more or less the same as above solution in execution but it will be mutch easier to maintain and work with the Logic App now.

The foreach now looks like: (removed the body to the function for clarity reasons)

"For_each": {
    "actions": {
        "INT0020-Update-Fuse-Users": {
            "inputs": {
                "body": {
                   ...
                },
                "function": {
                    "id": "/subscriptions/fakeee-15f5-4c85-bb3e-1e108dc79b00/resourceGroups/rgroup/providers/Microsoft.Web/sites/appname/functions/INT0020-Update-Fuse-Users-Compare"
                }
            },
            "runAfter": {},
            "type": "Function"
        }
    },
    "foreach": "@body('INT0020-Split-Array')",
    "runAfter": {
        "GET_FUSE_TOKEN": [
            "Succeeded"
        ]
    },
    "runtimeConfiguration": {
        "concurrency": {
            "repetitions": 2
        }
    },
    "type": "Foreach"
},

Logic App in the designer will now look like:

Triggers also have concurent control

We can also set this on triggers, sample bellow is on a recurrent trigger (or polling trigger withou spliton) will look like;

"trigger": {
"type":"http",
"recurrence": {},
"runtimeConfiguration": {
    "concurrency" : {
    "runs": 10
	}
}

For more detailed explanation on Trigger handling I suggest you read this great post from Toon Vanhoutte

I’m really happy that the product group finally has released this since it’s such an important part in our new world of ditributed landscapes and our “new” responsibility and interest in making sure our solutions are not flooding the destionations. We got the power and now we got the possibility to restrain them to the appropiate amount.

OMS and Non Events

2017-08-27T00:00:00+00:00

When it comes to monitoring a integration flows, there are several types of monitoring that we need to cover, just the other day I got the requirement to make sure that a Logic App has been run atleast one time during a 24 hour period.

To solve this I started to look at possible solutions and turning my head to the Alert section of Logic Apps Diagnostics since it has the possibility to create a alarm based on input as number of runs, number of failed runs etc. unfortunally for me the max time limit was 6 hours. In this solution we are using Log Analytics and OMS as monitor tool, using the new Logic Apps Gallery it’s super nice (guess I need another post on this later on). Anyway there is alarm functionality inside the OMS Portal so I started to look in to that.

Alarms in OMS is easily created based on a Search so first of we need to create a search, this is done in the search area and easiest way is to click your way to a start of the search and then change the last parts manually, I needed to check that a workflow was executed and ended sucessfully. So my query ended up like this: (easy to reuse, just change the resource_workflowName_s to the name of your Logic App.

search * | where ( Type == "AzureDiagnostics" ) | where OperationName == "Microsoft.Logic/workflows/workflowRunCompleted" | where status_s == "Succeeded"   | where ( resource_workflowName_s == "INT002_Update_Work_Order2" )  

This query will return the results of succesfully runs, and we will be able to use it in our alarm.

Setting up the alarm is rather simple, just Add Alert Rule and fill in some information about the alarm, the important parts are the Search Query here you shall choose the saved query above, Time Window this is how far back we will look, so for our case we look 24 hours back. Alert Frequency is another time slot and it’s how often to check this rule, based on our case we wanted to check every hour. But in order for us to not get a new alert every hour (seems unnessisary to get 7 alarm emails during th night we can also specify Spuress alerts this will make sure that an alarm will not be sent out more often than every 6 hours.

So in reality we can now be sure that at most we will be alerted within 25 hours from the last run that there has not been any more runs and we will be alerted again every 6 hours until we have sucessfully completed a run.

There is possibility to invoke webhooks, Runbook or ITSM actions but in our case an email is good enough so we will use that. The email sent out is not containing that mutch information but it’s enought for us, we now that the flow has not been working and most likely the sending system has failed to send their file.

The email:

Monitoring for events that should happen but are not happening is always tricky and has historically always been hard to solve, so often collegues or other people in your organisation creates tasks for them to do daily checks of some sort. It can be small things like checking that a folder is empty or a log for a row with todays date on it. Even if the task itself is easy it burdens the people doing it and as more of these “small” tasks are added the more time it consumes, time these peoople should have spent working on more important tasks. So to be able to handle monitoring of predicted events and get an alarm we can give services of great value to these people, now we notify them if anything is stopped or so and they can focus on their work. When they can rely on the things we build, we are not only building robust integrations we are then also building trust and a service of great value.

It’s often the small things that makes the most.

New blog

2017-08-26T00:00:00+00:00

Now I thought it was time to create my own blog.

As you might now I’ve been blogging alot on my work blog, http://blog.ibiz-solutions.se/.

So now I’ve felt that I’m ready for my own blog. I’ll keep with the same line, bloggin about Integration and most part will most surely be about Integration in Azure.

I’ll cover updates on the source projects I contriubute to here i.e Logic App Template Creator and some other new fun projects, so stay tuned and poke me if I’m keeping a to low pase on my blog.

Hope to see you around!

Cheers!

Mattias Lögdberg

MSI with Service Bus

Removing the access SaaS key

Troubelshooting

Summary

MSI what and why?

Is this not what we accomplished before?

But isn’t this making my development process a pain?

What should I think off

Read more:

Using MSI in APIM to Cosmos

Troubelshooting tips:

Request blocked by Auth

Partition key 1 is invalid

Summary:

AAD Personal Token in requests with VS Code

Summary:

Oauth Token generation with Postman

Summary:

Troubelshooting Azure API Management

Built in Analytics

Application Insights

Live Debugging in VSCode

Summary:

API Management and Auth0 Consumer Friendly API's

Summary:

RBAC with API Management

Summary:

Setup Auth0 with API Management

Summary:

Azure Functions logs in Application Insights part 2

Connecting custom logs from TelemetryClient

Adding an Extra “Operation”

Summary:

Azure Functions logs in Application Insights

Logs in Application Insights

Create a log scope with BeginScope

Add Filterable Property to Request

Summary:

August 2019 update Logic App Template Creator

Updates

Thanks to contributors!

Resource Group Deployments 800 Limit fix

Summary:

Fixing Function v2 add extension issues in portal

Summary:

Logic App Template Creator Updates Januari 2019

Power Shell Gallery

Storage Connector Tables and Queues

New CommandLet Get-IntegrationAccountMapTemplate

Summary:

API Management Template Creator Updates December 2018

Power Shell Gallery

Split the Template to a template per resource

Summary:

Innovation enabled by Integration

Architecture that enables innovation?

Summary:

API Management Template Creator Updates August 2018

Functions support added

Updated Logic App Support

Trigger name fix

Add Logic App as operation to exisitng API fix

Removed some name parameters

Summary:

API Management Template Creator Updates May 2018

Support for Versions, with Version Sets

Support for Schema

Summary:

Access AAD Secured Web API's from API Management

Enabling the built in security Authentication and Authorization to the API’s

Provide access to the API

API Management Policy for Aquiring JWT Token

Summary:

Logic App Custom Connector WSDL Tips

Import Error

Runtime problems, unexpected behaviors and errors from the backend system

Summary

CI with Logic App Custom Connector

Januari update Logic App Template Creator