Introduction to BimlScript

gravatar

Scott Currie

Introduction to BimlScript

published 08.14.15

last updated 02.18.16


Part of lesson Introduction to BimlScript.

Share

Introduction

In this lesson, we will learn about how we can automate our Biml using BimlScript code nuggets. This is the most powerful, flexible, and integrated way automate Biml that should meet all of your automation needs.

Please note that we are going to use a minimal amount of C# code to illustrate the key BimlScript code nugget types. If you're not familiar with C# syntax, don't worry if everything doesn't make sense at first. We have a full C# primer coming up later in the lesson plan. However, I felt it was a good idea to give you an idea of how code nuggets work first - rather than front loading the lesson plan with a bunch of C# syntax instruction without any context for how it will be used in BimlScript.

Code Nuggets in General

The approach that BimlScript takes is to embed nuggets of code directly into your Biml XML. If you are familiar with web technologies such as ASP.NET, PHP, Ruby on Rails, ColdFusion, or similar technologies - we're taking the exact same approach. The only difference is that we're adding our nuggets to Biml rather than to HTML.

Let's start with a very brief overview example. Suppose we start with a Biml file containing a single package:

<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <Packages>
        <Package Name="Package1">
            <Tasks>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput>SELECT 1 AS Temp</DirectInput>
                </ExecuteSQL>
            </Tasks>
        </Package>
    </Packages>
</Biml>

Just a simple package that runs a single SQL statement that does nothing. What BimlScript code nuggets allow us to do is to start to implement variable replacement and automation.

<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <Packages>
        <Package Name="Package1">
            <Tasks>
                <# foreach (var query in queries) { #>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput><#=query#></DirectInput>
                </ExecuteSQL>
                <# } #>
            </Tasks>
        </Package>
    </Packages>
</Biml>

The above code sample won't actually build, because we are using variables that aren't defined. But it does illustrate important concepts. Notice how we have added nuggets of code that are delimited with <# ... #> and <#= ... #>. Each of those code nuggets contains a C# (or VB) expression or statement that is executed like a program. The output of that program is flat Biml XML. For example with some setting of the "queries" collection above, we might get the following output Biml XML:

<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <Packages>
        <Package Name="Package1">
            <Tasks>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput>SELECT 1 AS Foo</DirectInput>
                </ExecuteSQL>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput>SELECT 2 AS Bar</DirectInput>
                </ExecuteSQL>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput>SELECT 3 AS Baz</DirectInput>
                </ExecuteSQL>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput>SELECT 4 AS Qux</DirectInput>
                </ExecuteSQL>
            </Tasks>
        </Package>
    </Packages>
</Biml>

Don't worry if that didn't fully make sense yet. We're going to see many more examples that should help illuminate what is going on.

There are 5 basic types of code nuggets, and in the follow sections, we'll take a more detailed look at each of them.

Control Nuggets

Perhaps the most commonly used type of code nuggets are what we refer to as "control nuggets." They do exactly what you'd guess from their name. They allow you (among other things) to control your path through the BimlScript file based on some conditional logic. Here are a few examples of places where you might use control nuggets:

  • To define some variables that you'll use later.
  • To access external data sources such as flat files, databases, excel files, or web services to retrieve data that you use to guide your code generation
  • To add conditional statements so that certain Biml fragments are only appear in specific scenarios.
  • To add loops to repeat Biml fragments multiple times (e.g. once for each item in a collection).

The syntax to create a control nugget is very simple. Start your code nugget with a <#. Then write some .NET code. Finish your code nugget with a #>. Going back to our first example, if you want to add a variable declaration to the beginning of the file, it looks like this:

<# var emitPackages = true; #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <Packages>
        <Package Name="Package1">
            <Tasks>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput>SELECT 1 AS Temp</DirectInput>
                </ExecuteSQL>
            </Tasks>
        </Package>
    </Packages>
</Biml> 

Sometimes, the control nugget is intended to create a loop or a conditional. In that case, we usually want to wrap that control nugget around a Biml fragment. To do this, we actually create TWO control nuggets. The first starts the conditional statement and the second finishes it. Let's take a look:

<# var emitPackages = true; #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <# if (emitPackages) { #>
    <Packages>
        <Package Name="Package1">
            <Tasks>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput>SELECT 1 AS Temp</DirectInput>
                </ExecuteSQL>
            </Tasks>
        </Package>
    </Packages>
    <# } #>
</Biml> 

Now that's interesting! Note how we have the open curly brace for the if statement in the first code nugget. By itself, that would be invalid syntax. Since the second code nugget has the close curly brace }, everything works as expected. If I set emitPackages to true, I get a package. If I set it to false, I don't.

It's a simple, and perhaps artificial example, but it illustrates the usage nicely. From here, you can put arbitrarily complex code into your control nuggets. Get data from databases, nest loops and conditions, or do anything else your business requirements dictate. You're in "control."

Text Nuggets

These Biml fragments surrounded by code nuggets seem pretty powerful for implementing logic, but is there a way I can customize the Biml fragments, rather than just conditionalizing or copying them? There is, and it's called the text nugget.

Text nuggets will evaluate whatever C# or VB expression they contain and then replace the text nugget with the string representation of that expression value. Text nuggets are written much like control nuggets. The only difference is the nugget start delimiter. Start the text nugget with <#= (note that equals sign that we added). Then write your expression. Finish the text nugget with a #>.

Let's enhance our sample above by moving the query into a variable and then using a text nugget to add it to our Biml fragment:

<# 
    var emitPackages = true; 
    var query = "SELECT 1 AS Temp";
#>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <# if (emitPackages) { #>
    <Packages>
        <Package Name="Package1">
            <Tasks>
                <ExecuteSQL Name="Run SQL" ConnectionName="Target">
                    <DirectInput><#=query#></DirectInput>
                </ExecuteSQL>
            </Tasks>
        </Package>
    </Packages>
    <# } #>
</Biml> 

There are two important things to note about this example. First, we added the text nugget to the ExecuteSQL DirectInput Biml (after having stored the query in a variable). The other is a nice side note about code nuggets in general. They are not limited to a simple line. Make them as big or as small as you like.

That's probably enough introduction to the text nugget for now. Just remember that this is only scratching the surface and that with text nuggets you can go much further. For example:

  • Use arbitrarily complex business logic in the expressions. Make that string lower case, get the string value directly from a database. It's up to you.
  • Use any type inside the text nugget. The BimlScript compiler will automatically convert it to a string for you.
  • Use as many text nuggets as you like. In the extreme case, the entire BimlScript file could be nothing but code nuggets.
  • Mix text nuggets with static text. For example, Name="Package <#=name#>" is completely valid.

Class Nuggets

As our BimlScripts become increasingly complex, sometimes we find that there is some reusable C# or VB business logic that we are repeating. The Biml design team hates repetition, so we also support the class nugget to enable you to move that reusable logic into methods (and other .NET structures) that you can call from multiple places in your BimlScript file.

This tends to be a more advanced feature that most Biml developers don't start using until they already have a solution in production, so we won't go into the complexities of all the things that you can do with this feature. However, I think it helps to see at least one example with a utility method.

Suppose that we have the example above, but we'd like to give the package a unique name every time we execute the BimlScript. We'd also like to give the ExecuteSQL task a unique name. To do that we can use the Guid.NewGuid() function built into the .NET framework. That's fine to start with, but GUIDs aren't particularly user friendly names, so we expect to revisit this code later to create more user friendly names. Since we're calling this logic from multiple locations, we would like to centralize the implementation of the unique name computation. That way, when we come back to change it later, we only need to change it in one place. To do that, we'll create a GetUniqueName utility method with our current GUID naming logic.

But how do I define a utility method? That's what class nuggets are for. Start your class nugget with a <#+. Then write some methods, classes, properties, or other reusuable .NET code. Finish your class nugget with #>.

Applying this to our example above, we have:

<# 
    var emitPackages = true; 
    var query = "SELECT 1 AS Temp";
#>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <# if (emitPackages) { #>
    <Packages>
        <Package Name="Package <#=GetUniqueName()#>">
            <Tasks>
                <ExecuteSQL Name="Run SQL <#=GetUniqueName()#>" ConnectionName="Target">
                    <DirectInput><#=query#></DirectInput>
                </ExecuteSQL>
            </Tasks>
        </Package>
    </Packages>
    <# } #>
</Biml>

<#+
string GetUniqueName() {
    return Guid.NewGuid().ToString();
}
#>

Note that the properties, methods, and classes you define in your BimlScript class code nugget can only be used in their containing BimlScript file. You cannot use these properties, methods, and classes from other files unless you also include them in those other files.

Comments

Sometimes you might want to make a note for future reference in your code file. Alternatively, you might wish to disable some BimlScript code without deleting it. BimlScript provides a multiline comment capability using the <#* and *#> delimiters.

Returning to our earlier example, we'll add a couple of comments:

<# 
    var emitPackages = true; 
    var query = "SELECT 1 AS Temp";
#>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <#* Check to see if packages should be emitted *#>
    <# if (emitPackages) { #>
    <Packages>
        <#* var unused = "comment me"; *#>
        <Package Name="Package <#=GetUniqueName()#>">
            <Tasks>
                <ExecuteSQL Name="Run SQL <#=GetUniqueName()#>" ConnectionName="Target">
                    <DirectInput><#=query#></DirectInput>
                </ExecuteSQL>
            </Tasks>
        </Package>
    </Packages>
    <# } #>
</Biml>

<#+
string GetUniqueName() {
    return Guid.NewGuid().ToString();
}
#>

Directives

Directives permit you to provide instructions to the Biml compiler. That might seem like an unusual thing to do, but it is in fact quite common. For example, if you want to reference a third party SDK to get your metadata (e.g. the SalesForce or Microsoft Dynamics CRM .NET SDK), you can do so with a directive. In fact, there are many different types of directives, each playing a special role in the compilation process. Let's first look at the assembly referencing scenario.

To start a directive, use the <#@ delimiter. Then specify the name of the directive. Then optionally specify a list of arguments that the directive supports. Finally, close the directive with the #> delimiter. Here is the example for assembly references:

<#@ assembly name="Sdk.dll" #>

A few things to note:

  • This directive, like most directives, can be placed anywhere in the Biml file.
  • The path to the assembly can be an absolute path, a path relative to the file, a path relative to the Mist/BIDSHelper installation directory, or a fully qualified name of an assembly stored in the Windows Global Assembly Cache (GAC).
  • You can specify more than one assembly directive if desired.

As noted earlier, there are a variety of directives that perform various tasks. While most of these are for more advanced scenarios, here is a list of the most commonly used directives with a brief description of each:

template

The template directive is a general purpose directive that permits you to set miscellaneous compiler options. It is optional, and by itself does nothing. It is only the individual options that affect the compilation. There are many options, but the only three you are likely to use are:

  • language: The .NET language that is being used in your BimlScript code nuggets. This can be C#, VB, or specific versions of either. Note that if you do not specify a language option, the default is C#.
  • tier: Specifies when the BimlScript file should be compiled. You'll see later how the order of BimlScript compilation can affect your output. A file with a lower tier value will always be compiled before a file with a higher tier value. Tier values can be any integer.
  • designerbimlpath: Mist provides quick info, autocomplete, and a variety of other Biml editor features that depend on the code file being properly structured. But if you are working on a file that contains a Biml fragment for use with an include directive or CallBimlScript (see below), the Biml editor will lack the proper context and therefore cannot provide correct information. This option allows you to specify the XML path from which the fragment in the current file will be used. This gives Mist the context it needs. For example, if you were providing a task fragment, your value for this option should be "Biml/Packages/Package/Tasks".

import

.NET code has a feature called namespaces that allow types with the same name to be grouped in naming containers that prevent name conflicts across libraries and vendors. Think of it like multi-part names in the database world. And just as with multi-part names, they can be tedious to type out explicitly. Consequently, .NET programmers will often list the namespaces that they intend to use within a given file so that they can omit the namespace each time they use a type. The import directive lets you reference a namespace that will be a default for your BimlScript file. If you are familiar with C# or VB, it is the equivalent of the C# using statement or the VB Imports statement. Options include:

  • namespace: The namespace to be imported

include

If you find yourself repeating Biml code fragments in multiple locations, it might be helpful to isolate that fragment in a separate file and reuse it. That is what the include directive does. When you include a file, the BimlScript compiler will replace the include directive with the full contents for the referenced file. It's just as if you had copied and pasted it yourself in the code editor. Options include

  • file: Specifies the path to the Biml fragment file to include. It can be an absolute path, a path relative to the current file, or a path relative to the Mist/BIDSHelper installation directive.

property

In addition to the include directive, BimlScript offers a second type of code reuse mechanism called CallBimlScript. Whereas include is like copy/paste, CallBimlScript is more like a NET method. And just like a .NET method, CallBimlScript permits you to pass arguments that can be used to customize the output of the called file. To declare the expected input parameters, the called file uses the property directive. Options include:

  • name: Specifies the name of the argument. This is the name by which it can be used in the called file.
  • type: This is the full name of the .NET type, including namespace.
  • required: Specifies if this argument is required or optional. Optional arguments must be at the end of the arguments list.

code

Over time, you will develop a small library of code that you use as helpers across all of your BimlScripts. Sometimes it is more convenient to isolate that code in its own .NET code file - outside of the BimlScript altogether. But if it is in its own file, how can we call it? We could compile it to a .NET DLL and then use the assembly directive to reference it, but that seems like a lot of unnecessary work. The code directive makes it easy to simply reference the file and then be able to use the code defined therein. Options include:

  • file: Specifies the path to the .NET code file. It can be an absolute path, a path relative to the current file, or a path relative to the Mist/BIDSHelper installation directive.

dependency

Mist has a great feature called Live BimlScript. It enables Mist to automatically track dependencies among files so that if you modify one file, only those files affected by the change are recompiled. This provides a high performance IDE that always shows you the fully up-to-date view of the results of your Biml code. In some cases, it is possible to use code nuggets to create file dependencies that cannot be automatically detected by Mist. In these cases, you can provide Mist with a hint about that dependency using this directive. Note that this is something that only advanced Biml framework scenarios require. Options include:

  • file: Specifies the path to the file on which the current Biml file depends. It can be an absolute path, a path relative to the current file, or a path relative to the Mist installation directive.

target

Biml offers an advanced feature called transformers, which enable you to easily modify existing Biml objects. The target directive specifies that the containing Biml file is a transformer and allows you to configure options for what kinds of Biml objects the transformer should modify and what types of modifications should be made. To learn more about transformers, you can read my transformers primer. Options include:

  • type: Specifies the Biml object type that this transformer should attempt to modify
  • mergemode: Specifies the type of transformation that can take place, such as LocalMerge or LocalReplace
  • exactmatch: Specifies if the type option should be an exact match or if subtypes of that type can also be transformed

annotation

Just as with individual Biml objects, it is possible to place annotation information on an entire Biml file. This is particularly useful when managing very large Biml frameworks. Options include:

  • annotationtype: Specifies the type of annotation such as documentation or tag
  • tag: If the annotation type is a tag, this specifies the value of that tag
  • text: Specifies the string value that should be stored within the Biml file annotation

extension

There is a packaging feature in Mist called Biml Bundles that enable third party developers to combine a large amount of Biml code into an easily reusable file. A feature of Biml Bundles is to provide extension points for users so that they can be customized with proprietary business logic and organization-specific or industry-specific patterns. The extension directive enables the user to declare that the containing Biml file is intended to be used as one of these Biml Bundle extensions. Options include:

  • bundle: Identifies the name of the Biml Bundle being extended
  • extensionpoint: Identifies the name of the specific extension point within the Biml Bundle that this file will bind to

Of course, if you forget about these or want to learn more, the Mist Biml editor gives you all of this information and more in the quick info tool tips as you write directives.

Conclusion

While this has only been a very brief overview of the code nuggets available for use in your BimlScript files, hopefully you are starting to see the possibilities for the tremendous level of automation and customization that BimlScript offers your data solutions. In the following lessons, we'll learn more about how to use our BimlScript code nuggets in real-world applications.

Finished?

Complete the lesson Introduction to BimlScript:

You are not authorized to comment. A verification email has been sent to your email address. Please verify your account.

Comments

gravatar

Possum

4:44am 10.27.15

Thanks Scott for all these intros. They are massively helpful!