Toying with Azure Functions – A Trade Log

Azure Infrastructure, Windows PowerShell
Toying with Azure Functions – A Trade Log post image

Serverless.

The best way for me to learn something is to use it, I’m hands-on. After being barraged by articles and hype around Azure Functions, I decided to try it out. But, I’m not a developer. All the new stuff that the cool kids are playing with today (IoT, Serverless, Bots, AI, etc.) requires real dev skills but that’s just not what I do. Sorry, not sorry. So here we go with PowerShell and Azure Functions. Warning: support for PowerShell is “experimental” and I found that I had to do a lot of workarounds.

The project

I’m going to use Azure Functions to generate a trade log. I’ve used all kinds of stock brokers (Schwab, Scottrade, Fidelity, etc.) but no platform gives you the flexibility and cost of Interactive Brokers ($IBKR). Most of my trades cost $1.00 but you get what you pay for. $IBKR doesn’t have a nice, rich, web interface that shows your trade history. Instead you have to run  your own reports and luckily there’s an API for it. My function is going to create a “up to date” log of the trades that have been executed (a lot of my trading is automated so I might not know when something has been bought or sold).

I’m going to focus on Functions and specifically this project. There’s a lot of great documentation on the Interactive Broker API as well as Azure Functions on their respective official documentation pages.

Setting up

So the first thing I’m going to do is create a new Azure Functions app using PowerShell and starting with the Timer trigger. Let’s call this “GetIBTrades2Blob.” We’re going to go right into the “Integrate” section and configure the following:

Triggers: Timer Trigger called “Daily0200ZZ” that has a schedule of “0 0 2 ***” That means, it will run at ANY day of week, month, or day at 2 hrs, 0 minutes, and 0 seconds. That’s 2:00:00 daily (Zulu time), which is 8 PM Eastern.

image

Inputs: Azure Blob Storage. I have a parameter name “inputBlob” with an explicit path, “trades/tradelog.csv” and a Storage account connection. Azure Functions sets this up nicely just by going through the portal UI. One note is that I struggled a lot with the path. It seems PowerShell doesn’t really support variables here (the default is {file}). I hope this will be fixed at some point but it may just be a limitation of the language.

image

Output: Azure Blob Storage. This is setup almost exactly the same way and the input, we’re just going to use a different parameter name but the same path and connection.

image

All of that just writes your functions.json file for you. Next, we need to write the code in run.ps1.

Code

The code isn’t really relevant for this blog post except for a few key pieces that make everything work. The code might as well just be something like

?View Code POWERSHELL
1
“hello world” | Out-File –FilePath $outputBlob

But, let’s see what we’re dealing with for Interactive Brokers. First, we need two parameters, a Token Code and a QueryID (see this link). This can be retrieved from your $IBKR account. For the QueryID, I created a new one (that is, a new Flex Query) that just gets my trades for the current day (see this link).

Now that we have those, we can get the trade confirmation report for today using Invoke-WebRequest and doing some parsing.

?View Code POWERSHELL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$token = "000000000000000000000000"
$q = "123456"
 
$request = "<a href="https://gdcdyn.interactivebrokers.com/Universal/servlet/FlexStatementService.SendRequest?t=">https://gdcdyn.interactivebrokers.com/Universal/servlet/FlexStatementService.SendRequest?t=</a>$token&amp;q=$q&amp;v=3"
 
$response = Invoke-WebRequest $request -UseBasicParsing
 
[xml]$xml = $response.Content
[string]$refCode = $xml.ChildNodes.ReferenceCode
[string]$flexUrl = $xml.ChildNodes.Url
 
$reqData = $flexUrl + "?q=$refCode&amp;t=$token&amp;v=3"
$responseData = Invoke-WebRequest $reqData -UseBasicParsing
   
$content = $responseData.Content

What will be returned here is an array containing the trades returned by the query, for example:

Date/Time,Symbol,Quantity,Price,Amount,Commission,OrderType,TradeDate
20180518;134156,XYZ,-100,5.2,-94,-1,LMT,20180518

Great! But now I need to add this trade to the existing ones (from yesterday, the day before, and before that, etc.).

This is a learning exercise so I decided to get the current days trades. I realize that there is a “Month to date” Flex Query available in $IBKR. One of the things that I learned through this exercise is that appending data is not easy (or possible) in Azure Functions using PowerShell. For C#, many examples exist using Append Blob (some information here on that). So, with that said, we have to do a workaround. The workaround will be to get the existing tradelog.csv and that’s exactly why we added the Azure Blob Storage as an input.

?View Code POWERSHELL
1
2
3
4
5
6
7
8
9
10
11
12
13
#Remove header information from $content since it will already be there in the file we’re getting
$cleanContent = $content | Select-Object -Skip 1
 
#Get the current file from blob storage
$inputArray = Get-Content -Path $inputBlob
 
foreach ($line in $cleanContent)
     {
         $inputArray += $line
     }
 
 
Out-File -Encoding ASCII -FilePath $outputBlob -inputObject $inputArray

This is pretty simple. First we remove the heading. Then we loop through each line in the returned results for today’s trade since there could have been more than one trade. The last line is where we store the full text (including the original input) back to Azure Blob Storage. Actually, by adding additional Outputs, you can save the file in many places. For my purposes, I also have the file going to my OneDrive.

In my example, I was working with text. Working with binaries or media would be much harder because of some content type issues, unless you use C# or another supported language. A big thanks to jschmitter, who patiently assisted me working all of this out.

0 comments

Using Azure Policy Sets

Azure Infrastructure

At Microsoft Ignite in September 2017, Ryan Jones (@rjmax) discussed Azure Resource Manager Policies and some enhancements coming soon. See this blog post about the public preview announcement. One of those, was Policy Sets. Policy Sets allow you to group several policies together and assign them as a group. There’s more information at http://aka.ms/azurepolicy. In this blog post, we’re going to explore how to start using them.

Start with existing policies

In order to create a policy set, we need existing policy definitions. You can use some built-in ones but for this example, I’ve created 4 custom policies which are related to storage:

  1. Audit VM’s that don’t use Managed Disks
  2. Deny deployment if Storage Account Blob Encryption is not enabled.
  3. Deny deployment if Storage Account File Encryption is not enabled.
  4. Deny deployment if Storage Account https-only transport (secure transfer required) is not enabled.

The first thing we need to do is get the Policy Definition Id’s. We need the Policy Definition Name to get those. Here, I’ve looked up the names and created an array of the Policy Definitions that I want in my new Policy Set.

?View Code POWERSHELL
1
2
3
4
5
$policyNames = @( "audit-managedDisks", 
"deny-NoBlobEncryption", 
"deny-NoFileEncryption", 
"deny-NoHttpsOnly" 
)

Next, I’m going to loop through those and get the Policy Definition Id and store it in a variable called $policyDefinitionId. Since I need to loop through each Policy Definition now, I’m also going to construct an object called $policySetDefinition which we’re going to use later.

?View Code POWERSHELL
1
2
3
4
5
6
7
8
9
$Target=@()
$policyDefinitionIds = @()
$policyNames | 
    ForEach-Object {
        $policyDefinitionId = (Get-AzureRmPolicyDefinition -Name $_ | select -ExpandProperty PolicyDefinitionId)
        $TargetObject = New-Object PSObject –Property @{policyDefinitionId=$policyDefinitionId}
        $Target +=  $TargetObject
    }
$policySetDefinition = $target | ConvertTo-Json

 

Define and Assign the Policy Set

Next we need to create the Policy Set Definition

?View Code POWERSHELL
1
2
3
4
5
6
7
$policySetParams = @{ 
Name = "policySet-Storage" 
DisplayName = "Storage: Policies to enhance security of Storage Accounts." 
Description = "This initiative contains several Storage Policies to be applied at the subscription level." 
PolicyDefinition = $policySetDefinition 
} 
$policySet =  New-AzureRmPolicySetDefinition @policySetParams -Verbose

Notice that this command takes the $policySetDefinition we created earlier into the PolicyDefinition parameter. Now that we’ve created a Policy Set Definition, we can assign it.

For this example, I’m going to assign it to my Subscription but I need to exclude two Resource Groups. With the new policy language, that’s pretty easy to do. We are also going to define the Sku. The Sku is an object consisting of a name and tier. I’m going with Standard here because I want to enforce this policy set on existing resources. If you only wanted to enforce the policy set on new resources, set this to A0 and Free.

?View Code POWERSHELL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ExcludedResourceGroup1 = Get-AzureRmResourceGroup -Name "rg-aad" 
$ExcludedResourceGroup2 = Get-AzureRmResourceGroup -Name "securitydata" 
$sku = @{ 
name="A1" 
tier="Standard" 
}
 
$policyAssignmentParams = @{ 
Name = "StoragePolicySetAssignment" 
DisplayName = "Storage Policy Set" 
Description = "This initiative contains several Storage Policies to be applied at the subscription level." 
PolicySetDefinition = $policySet 
Scope = "/subscriptions/{guid-of-subscription}" 
NotScope = $ExcludedResourceGroup1.ResourceId, $ExcludedResourceGroup2.ResourceId 
Sku = $sku 
} 
$new = New-AzureRmPolicyAssignment @policyAssignmentParams

Visualize

That’s it! Let’s take a look at how this appears in the new Policy UI. Here’s the Assignments blade:

image

And here’s the actual Assignment showing the scope, exclusions, and sku:

image

1 comment

Trusted Sites With IE ESC Turned On

Enterprise, Microsoft

When protecting users from malicious internet sites on servers, we can keep Internet Explorer Enhanced Security Configuration turned on (that’s the default). However, with the setting on, almost no modern website will properly load.

The solution is to add these sites you trust to your Trusted Sites zone. In an enterprise environment, we would leverage Active Directory Group Policy to do this. The way to add specific sites to a zone is well-documented. In short, we use the Sites to Zone Assignment List policy.

However, this doesn’t work with IE ESC turned on. This KB article hints at why. Although it says it applies to Windows Server 2003, I’m working with Windows Server 2016. My machine will be a Remote Desktop host and I want to lock it down. So, how can we keep IE ESC on and allow a specific list of sites to have looser security settings?

It seems there’s two ways this could work. In both cases, we still configure the Sites to Zone Assignment List:

1. Apply the Method 2 workaround listed in the KB article above. You may have to create the keys that are not present.

image

2. This one is much more complicated. We will configure the EscDomains registry key. This key is described in a support article:

The EscDomains key resembles the Domains key except that the EscDomains key applies to those protocols that are affected by the Enhanced Security Configuration (ESC). ESC is introduced in Microsoft Windows Server 2003.

We can use Group Policy Registry Settings to update the registry. Enter the values in HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\ESCDomains.  Once completed, it will look something like the below:

Group Policy view:

image

Registry view:

image

Regardless of method, you can load up one of your sites and click File, Properties. The Zone should show “Trusted Sites.”

image

One note, this machine also has other Group Policies applied from the Microsoft Security Guidance blog and other policies may change the behavior. Is there a better way? I don’t know, you tell me, comments below.

1 comment

Managed Storage Account SAS Tokens

Azure Infrastructure

Introduction

Building on the previous blog post where we configured Azure Key Vault to automatically rotate Storage Account Keys, this post will discuss SAS tokens (Shared Access Signatures). As a quick refresher, using SAS tokens is the recommended way to interact with your Storage Account. For more information see Using shared access signatures.

Create SAS Definition

In order to create a SAS definition, you will need the setsas permission. We can add this to the list of permissions we used in the previous blog post:

?View Code POWERSHELL
1
Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ResourceGroupName $keyVaultResourceGroupName -UserPrincipalName $upn -PermissionsToStorage set, get, regeneratekey, setsas

Now, we can create a SAS definition. For my example, I want to:

  • Limit the SAS definition the Blob service (and not Tables, Queues, or Files).
  • Name it sas1
  • Limit to https only
  • Limit to only my current IP address (IP whitelist)
  • Limit the validity of the token to 5 days
  • Limit the permissions to Read and Write.

In the command below, I’m first getting my IP address using ipinfo.io site. Then I’m using the Set-AzureKeyVaultManagedStorageSasDefinition to create a new definition.

?View Code POWERSHELL
1
2
3
4
5
6
7
8
9
10
11
$ip = Invoke-RestMethod http://ipinfo.io/json | Select -ExpandProperty ip
$sasDefinition = Set-AzureKeyVaultManagedStorageSasDefinition `
-Service Blob `
-ResourceType Container,Object `
-VaultName $keyVaultName `
-AccountName $storageAccountName `
-Name 'sas1' `
-Protocol HttpsOnly `
-IPAddressOrRange $ip `
-ValidityPeriod ([System.Timespan]::FromDays(5)) `
-Permission Write,Read

Once you do this, you’ll see a new secret in your Key Vault. Now, let’s get the secret value.

?View Code POWERSHELL
1
2
$secret = Get-AzureKeyVaultSecret -VaultName $keyVaultName -Name ($sasDefinition.sid).Split("/")[-1]
$sasToken = $secret.SecretValueText

Use SAS Definition

That’s it! Now, let’s test this by uploading a file. There’s nothing new here, I’m simply using Set-AzureStorageBlobContent with a context. The context is generated from the SAS token we retrieved in the previous step.

?View Code POWERSHELL
1
2
3
4
5
$container = "docs"
$localFile = "C:\Temp\FUNDAMENTALS OF AZURE 2ND ED.pdf"
$blobName = "Fundamentals of Azure.pdf"
$ctx = New-AzureStorageContext -SasToken $sasToken -StorageAccountName $storageAccountName
Set-AzureStorageBlobContent -File $localFile -Container $container -Blob $blobName -Context $ctx –Verbose

There could be many use cases for this. For example, if several users need to upload files to blob storage, you can generate a unique SAS for each one. Another example could be an application querying Key Vault to get a SAS token. This one is documented on the official documentation here.

0 comments

Azure Storage Account Keys Automatic Rotation

Azure Infrastructure

Introduction

A new feature called Managed Storage Account Keys popped up last month that promises to make rotating Azure Storage Account Keys easier. I have to give a ton of credit to my colleague Dmitriy S. who helped with a lot of the discovery and troubleshooting.

Why rotate?

It is recommended to periodically rotate your Storage Account Keys. Storage Account Keys give the holder full access to your storage account and all data within it. Rotating the keys periodically help mitigate the risk of unauthorized access.

The old way

Previously, we had this article called “Set up Azure Key Vault with end-to-end key rotation and auditing,” which describes automation storage account key rotation. While that will still work, it’s very complex to setup, involves using an Automation Account and Runbook, Functions, and a Logic App. We now have a much easier way!

The new way

Using Managed Storage Account Keys, we can easily set up automatic key rotation by just deploying a Storage Account and a Key Vault. The article called Azure Key Vault Storage Account Keys describes the feature and developer experience. However, the “Getting started” section was a bit confusing, incomplete and it looks like it contains some errors. So, let’s go through it together.

Set-up permissions

I’m going to assume you have a Key Vault and Storage Account already and they have registered providers. The first thing we need is the ObjectId of Key Vault. Note, we need the ObjectId of the Key Vault service (“first party identity”), NOT your specific Key Vault. This object ID is different for each tenant. The instructions on the official documentation say:

?View Code POWERSHELL
1
Get-AzureRmADServicePrincipal -SearchString "AzureKeyVault"

However, that didn’t work for one of my tenants (which was a Enterprise Enrollment). A more accurate way to get the ObjectId is to first login using “Connect-AzureAD” and then using the following command:

?View Code POWERSHELL
1
$servicePrincipal = Get-AzureADServicePrincipal -Filter "AppId eq 'cfa8b339-82a2-471a-a3c9-0fc0be7a4093'"

This won’t work on MSDN Azure subscriptions, but neither did the first way. I’ve found that the AppId above is consistent across all of my tenants but could be different for AzureChina, AzureGov, etc. Next, we need to assign the “Storage Account Key Operator Service Role” role which gives the Key Vault list and regenerate permissions:

?View Code POWERSHELL
1
2
$storageAccountId = (Get-AzureRmStorageAccount -StorageAccountName wspladiag02 -ResourceGroupName wspla-rg).Id
New-AzureRmRoleAssignment -ObjectId $servicePrincipal.ObjectId -RoleDefinitionName 'Storage Account Key Operator Service Role' -Scope $storageAccountId

Storage account onboarding

The official documentation is unclear on the next steps, in my opinion. But, here’s how we set up daily key rotation using PowerShell. First, we need to make sure we (user performing operation) has “set” operation allowed within the Key Vault’s access policies. If you don’t, you’ll get this error later on:

?View Code POWERSHELL
1
Add-AzureKeyVaultManagedStorageAccount : Operation "set" is not allowed by vault policy

So to prevent that, let’s add the current user:

?View Code POWERSHELL
1
2
3
$keyVaultName = "kv-infra-secrets"
$keyVaultResourceGroupName = "RG-Infrastructure"
Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ResourceGroupName $keyVaultResourceGroupName -UserPrincipalName "wahid@contoso.com" -PermissionsToStorage set, get, regeneratekey

I gave the user get and regeneratekey also but those aren’t required unless you want to use other commands. Now let’s create a TimeSpan object:

?View Code POWERSHELL
1
$regenerationPeriod = [System.Timespan]::FromDays(1)

Next, we use a new command to add the Storage Account as “Managed” by this Key Vault. If you don’t have this command, remember to update to the latest Azure PowerShell.

?View Code POWERSHELL
1
2
3
4
5
6
7
8
$parameters = @{
VaultName = $keyVaultName
AccountResourceId = $storageAccountId
AccountName = "mystoragetest1"
ActiveKeyName = "key1"
RegenerationPeriod = $regenerationPeriod
}
Add-AzureKeyVaultManagedStorageAccount @parameters

That’s it! The Storage Account Key will be rotated on the schedule you set up. You can verify everything by using Get-AzureKeyVaultManagedStorageAccount. You can also regenerate the keys manually using Update-AzureKeyVaultManagedStorageAccountKey if you don’t want to wait.

4 comments