Manage configuration with Estrelica.Core.Configuration

While it is possible to handle the Archer connection process via hard-coded values and direct calls to Core.ValidateLicense() and Core.CreateSession(), it is recommended to store all of your settings in configuration files, and let the CoreConfig.Load() class handle the heavy lifting for you. This is included in the Estrelica.Core.Configuration package available via NuGet.

If you follow the convention described below in creating a JSON appSettings file, you can simply call the CoreConfig.Load() method and it will handle all of the actions above. This allows you to manage your configuration separately from your code, while also supporting various alternate configuration and override behaviors.

Here's an example appSettings.json file for the configuration values described on the Connecting To Archer page:

{
    "CHSAuthKey": "5E56F79E-E678-4BAF-8FCF-B50141FAD5C7",
    "Archer": {
        "url": "https://archer.company.com",
        "instance": "Archer-Prod",
        "username": "apiuser@company.com",
        "password": "Password123",
        "connectionString": ""
    }
}

If this configuration is stored in a file named appSettings.json in the application's current operating directory (typically the same directory where the application executable resides), you can simply call CoreConfig.Load() with just your (required) warningCallback and it will take care of everything. It will validate your license (assuming you've entered a valid "CHSAuthKey"), authenticate with Archer (if all of your "Archer" settings are correct), and return a Core instance ready for use:

var core = Estrelica.CoreConfig.Load(wm => Log(wm.Message));

Just like when calling Core.Validate() directly as shown in the Connecting to Archer article, the Action<Exception> warningCallback parameter is required. This is how Estrelica.Core will let you know of any non-fatal issues arising during the license validation step, including notice of upcoming expiration. The CoreConfig.Load() method will also use this callback to let you know of any problems it finds while processing your settings.

If your configuration is stored in a different location or has a different filename, you can override the default expectation via the appConfigFilename parameter:

var core = Estrelica.CoreConfig.Load(wm => Log(wm.Message), appConfigFilename: "c:\temp\alternateSettings.json");

As the example above shows, it may be tempting to use alternate filenames during development to override the actual appSettings.json file with alternate settings. For example, the appSettings.json file in your project directory may contain production configuration, intended for use when the app is actually deployed, but during development you'll most likely want to use a different Archer server, instance and/or credentials. Handling this scenario by specifying an alternate filename is a bad idea, however, as it leads to the risk of accidentally releasing code to production with that alternate dev-only expectation included.

User Secrets

For these "alternate configuration" scenarios, the CoreConfig.Load() method supports user secrets. This allows each developer on a project to maintain their own local app settings, independent of source control and available only on their local development machine. This allows not only separation of dev versus prod configuration, but also separation of individual developers' credentials from one another. Furthermore, user secrets are not limited to development. You may choose to use them in production as well, so that actual production credentials need not be stored with the project in source control. Since they are maintained separately from the application this also eliminates the risk of overwriting them during an upgrade deployment.

Microsoft provides a detailed writeup of user secrets in their documentation, but that information is overly complicated so don't be put off by it. Think of user secrets as just another file (named secrets.json) which resides outside of your source tree in a particular location on each developer's machine, following the format of your appSettings.json file. There's no need to use the Secret Manager tool to manage your user secrets as described in that article, as any text editor (e.g. the Visual Studio IDE) can modify the secrets.json file. Microsoft's documentation also implies that user secrets only apply to ASP.NET applications, but in fact they can be used with any type of .NET application which references the Microsoft.Extensions.Configuration.UserSecrets assembly.

If user secrets are provided, any settings specified in the secrets.json file will override the corresponding values in appSettings.json. The secrets.json can include all values, or only those which are to be overridden.

During the configuration load process the two files are effectively merged by first loading the appSettings.json file (if present), then replacing or inserting values into the loaded configuration with any same-named values found in the user secrets file (if present).

While both files are optional (allowing you to manage your entire configuration in user secrets, if you prefer) at least one must be present. If neither file exists, an exception will be raised. Futhermore, all required values (e.g. the CastleHill Software authentication key and Archer connection/credential info) must be present in the merged result. CoreConfig doesn't care where each value comes from (either appSettings.json or secrets.json), but they must be in at least one of the two files.

How do I create and manage the user secrets file?

Implementing and managing user secrets in Visual Studio is fairly straightforward. Simply right-click on your project and choose "Manage User Secrets". This will create a new empty file in the local user's AppData folder (on their local filesystem, outside the project itself) named secrets.json, then automatically open it for you to edit in the Visual Studio editor.

Note: The first time you manage user secrets on a project, you may be presented with this dialog:

followed by an error like this:

If so, you can resolve this problem by manually installing the latest version of Microsoft.Extensions.Configuration.UserSecrets from nuget.org via the NuGet Package Manager, then click "Manage User Secrets" again.

ref: https://developercommunity.visualstudio.com/t/visual-studio-2019-tries-to-install-wrong-nuget-pa/847169

Each user secrets file has its own Id (which is actually just the short name of the directory in the user's AppData folder where the secrets.json file is stored). When creating a user secrets file in Visual Studio, the Id is system-generated and can be discovered by opening the project's csproj file and examining the <UserSecretsId> node:

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <UserSecretsId>1f5e7a05-94fb-44be-b10f-ff9e17ccbc55</UserSecretsId>
  </PropertyGroup>

When you build your project, the .NET compiler will automatically add a UserSecretsIdAttribute to your assembly, carrying the Id found in the node of your csproj file. At runtime, the CoreConfig.Load() method will parse your assembly for that UserSecretsIdAttribute. If that Id references a valid file, it will merge those user secrets with the appSettings file as described above.

The user secrets for your project are stored in a file located at "%appdata%\Microsoft\UserSecrets\xxxx\secrets.json", where "xxxx" is the User Secrets Id for the current project. This folder is not part of the Visual Studio solution, ensuring that whatever secrets are stored in the file will never get committed to source control.

User secrets also work under Linux, but they can't be managed via the VS Code IDE quite as easily as with Visual Studio under Windows. You'll need to manage the secrets.json file yourself, and ensure that it is stored in the "~/.microsoft/usersecrets/xxxx/" directory of the executing user account, where "xxxx" is the User Secrets Id for the current project.

It's worth noting that the secrets.json file is just a plaintext JSON file, so you can copy it from machine to machine or edit it with any text editor as needed.

How does a user secrets file affect CoreConfig.Load()?

Estrelica.Core refers to user secrets by their Ids, not by their full file path.

When you call CoreConfig.Load(), user secrets are considered in four ways:

  1. If you explicitly pass a non-null value to the "userSecretsId" parameter of that method, the configuration loader will use that secrets Id.
  2. Otherwise, if your project has a UserSecretsIdAttribute (an automatic side-effect of using the "Manage User Secrets" approach described above) the configuration loader will assume to use that attribute's secrets Id.
  3. If no user secrets Id is identified by either of the above cases, the configuration loader will assume to use "Estrelica.Core" as the user secrets Id. This allows you to just create and maintain a single common "Estrelica.Core" secrets.json file on your machine, and never have to add/modify in any of your projects. (You will, however, lose the right-click "Manage User Secrets" capability in Visual Studio).
  4. If String.Empty (or "") is passed to the "userSecretsId" parameter, all of the user secrets logic will be skipped and no user secrets will be loaded. Only the configuration found in your appSettings.json file will be considered.

If a user secrets Id was identified by case 1 or 2 but no secrets.json file was found for that Id, the CoreConfig.Load() method will issue a warning via the warningCallback action. This is intended to address cases where, for example, all the settings needed for a production deployment are in the appSettings file, but during development you want to override some of those settings with local user secrets. If you deploy your code to production with the user secrets Id (whether by explicit parameter or a UserSecretsIdAttribute) intact, and no user secrets file exists on the production server under that Id, this will only result in a warning. The production application will find everything it needs in the appSettings file alone.

The CoreConfig.Load() method does not issue a warning if no user secrets file is found for case 3, however, since the caller didn't specifically ask that "Estrelica.Core" be considered.

In all three cases (1, 2, and 3), if the Id resolves to a secrets.json file, a message will be logged indicating that the user secrets file was found.

After all the above, if a valid secrets.json file is identified, it will be merged with appSettings.json (if present) before proceeding to retrieve the values needed to continue.

Any and all settings in the appSettings.json file may be overridden via secrets.json, as long as the same format is followed. In fact, if secrets.json contains everything that's needed (the CHSAuthKey and all of the Archer settings), the appSettings.json file itself becomes optional, and the CoreConfig.Load() method won't throw an exception if appSettings.json is not found.

Let's see a real-world example

Consider the scenario where we have the "production" appSettings.json file shown at the beginning of this article as part of our project (shared via source control with everyone on the team, and included in the deployment package intended for the production server). That file contains all the settings needed by the application when it runs in production. During development, however, we want to target a development Archer instance on a different server. Otherwise, the rest of the settings (our CHSAuthKey and user credentials) are the same. Only the URL and Archer instance name need to be overridden.

We've right-clicked on the project and chosen "Manage User Secrets", which opens a file named secrets.json from our local AppData folder in the Visual Studio IDE. If we're new to the team, this will be an empty file, so we'll paste and save this information into it:

{
    "Archer": {
        "url": "https://dev-archer.company.com",
        "instance": "Archer-Dev"
    }
}

In this example, while we're executing the application during development, Estrelica.Core will authenticate with Archer as before, using the settings defined in the default appSettings.json file, but it will selectively override the values in that configuration with any found in the secrets.json file. In other words, it will still use the apiUser@company.com/Password123 credentials from appSettings.json, but it will authenticate instead with the Archer-Dev instance on the dev-archer.company.com server, as specified by the overrides found in the local user's secrets.json file.

When the application is complete and ready for QA, the QA team can create their own local secrets.json files directing the application to communicate with their Archer test instance. They might even add additional settings for "username" and "password" to test how the application runs under different user contexts.

Why should I use user secrets?

It's important to understand that, by design, this secrets.json file is NOT part of the Visual Studio project, and therefore will not be committed to source control. It exists only on the local user's system. This provides security for each developer's settings, but also means that any other developers working on the same project MUST also go through the right-click -> "Manage User Secrets" process above in order to manage their own local configuration. However, the <UserSecretsId> node in the csproj file WILL be committed to source control, so that Id and the location of the secrets.json file will remain consistent across the team.

I don't like that Guid Id in the example above, how can I make it more memorable?

As shown in the example above, the user secrets Id is an auto-generated Guid, created by Visual Studio the first time "Manage User Secrets" is selected on a project. This action takes care of inserting a new <UserSecretsId> node into the csproj file, generating a unique Id, then creating the directory structure and the file itself on the local filesystem. This Id need not be a Guid, however. The only reason Visual Studio defaults this to a Guid is because it's an easy to ensure uniqueness, but it doesn't really need to be unique. You might want to share the same secrets.json file across multiple projects, so changing it to some other, more memorable value is a good idea.

To change the user secrets Id,

  • Close any editors that currently have the secrets.json file open.
  • Open the project's csproj file and take note of the current <UserSecretsId> value.
  • Open File Explorer and navigate to "%appdata%\Microsoft\UserSecrets". There you will see a directory having the same name (i.e. a Guid) as the <UserSecretsId> value from the previous step. This is the local directory where the secrets.json file resides.
  • Rename this directory with a valid name of your choosing. This will become your new user secrets Id. A good example might be "Estrelica.Core" (which is assumed as the default user secrets Id on all of your projects if no other action is taken -- see above).
  • Return to the csproj file and paste the new directory name (e.g. "Estrelica.Core") into the <UserSecretsId> node. Save the csproj file.
  • Right-click the project and choose "Manage User Secrets". This will re-open the secrets.json file from its new (renamed) location in the Visual Studio editor. Confirm that its contents are as expected.

Environment overrides

When implementing an integration project it's not uncommon to have to deal with multiple Archer instances, potentially hosted on multiple web servers. The examples above show how an appSettings.json file and/or user secrets.json file may be used to connect Estrelica.Core to a single Archer instance, but what if you have several?

One way to handle this would be to create separate appSettings.json and/or secrets.json files for each, and selectively pass their filenames and/or Ids into the CoreConfig.Load() method as needed when switching between instances. This is cumbersome however, and can lead to runtime errors or unexpected results if the wrong user secrets get merged into your app settings.

Estrelica.Core.Configuration handles this by supporting a second level of override behavior. This is conceptionally orthogonal to the appSettings -> user secrets overrides discussed above. Those allow settings in one file to replace those in another. Here we're talking about a way to override settings within either of those files with alternate settings in the same file.

To do this, you just need to create a sub-node under the "Archer" node, and nest your override settings within it, then reference that sub-node's name during the call to CoreConfig.Load() in the "configOverrideKey" parameter.

Expanding on the above example, imagine a scenario where you have two Archer instances (named "Archer-Prod" and "Archer-UAT") running on the same Archer server (at https://archer.company.com). Furthermore, you have the same user (apiuser@company.com) in both instances, but with different passwords in each.

This scenario can be handled by nesting the necessary overrides directly inside the Archer node as follows.

We'll start by moving the two values that differ between the environments (instance and password) down into a new override node named "Production", nested under the standard "Archer" node in appSettings.json:

{
    "CHSAuthKey": "5E56F79E-E678-4BAF-8FCF-B50141FAD5C7",
    "Archer": {
        "url": "https://archer.company.com",
        "username": "apiuser@company.com",
        "connectionString": "",
        "Production": {
            "instance": "Archer-Prod",
            "password": "Password123"
        }
    }
}

Then we'll add another override node named "UAT" and put its values for those same two settings there:

{
    "CHSAuthKey": "5E56F79E-E678-4BAF-8FCF-B50141FAD5C7",
    "Archer": {
        "url": "https://archer.company.com",
        "username": "apiuser@company.com",
        "connectionString": "",
        "Production": {
            "instance": "Archer-Prod",
            "password": "Password123"
        },
        "UAT": {
            "instance": "Archer-UAT",
            "password": "v)SE(*Fh2kj;alk"
        }
    }
}

The "key" names used in this example ("Production" and "UAT") for the override sub-sections are arbitrary, so you may define them with whatever names make sense for your needs. The only requirements are that they must be valid JSON key strings, must be unique within the "Archer" node, and must not collide with any of the standard keys (url, instance, username, password or connectionString) in the "Archer" section.

With this type of configuration you can use the optional "configOverrideKey" parameter on the CoreConfig.Load() method to tell it which environment-specific overrides should be used. For example, calling

var core = Estrelica.CoreConfig.Load(wm => Log(wm.Message), 
    configOverrideKey: "UAT");

will cause Estrelica.Core to authenticate with the "Archer-UAT" instance running on the same server as before, using the same user account (apiuser@company.com) but with "v)SE(*Fh2kj;alk" as the password rather than "Password123".

In other words, the base config will be loaded from the standard "Archer" key, then, if the "configOverrideKey" parameter is not null, and a sub node exists inside the "Archer" node having that value as its key, the values found inside that sub node will be "promoted" up to the Archer node, overriding any values it contains.

This is ideal for situations where you have multiple target environments which share one or more of these values. For example, if you have Production, UAT and Development installations of Archer, all of which have the same "XYZ" instance name and identical user credentials, but are running on three different IIS servers, you might use a configuration like this:

{
    "CHSAuthKey": "5E56F79E-E678-4BAF-8FCF-B50141FAD5C7",
    "Archer": {
        "instance": "XYZ123",
        "username": "apiuser@company.com",
        "password": "v)SE(*Fh2kj;alk",
        "Production": {
            "url": "https://archer.company.com"
        },
        "UAT": {
            "url": "https://archer-uat.company.com"
        },
        "Dev": {
            "url": "http://devserver.local"
        }
    }
}

Then you'd simply need to specify one of the three configOverrideKey values ("Production", "UAT" or "Development") during the call to CoreConfig.Load() in order to select which installation Estrelica.Core will communicate with. (Note that in this scenario, since the "url" is only specified in the override nodes, a configOverrideKey must be provided to the Load() method. A null (default) configOverrideKey will result in an exception since there is no "url" specified under the default "Archer" node, so Estrelica.Core will not know which URL to use for its communication.)

Furthermore, these overrides may be combined with the user secrets overrides as well. This allows for team development scenarios where each team member is expected to use their own personal Archer account(s) for development and testing against multiple Archer instances. For example, you may choose to remove all usernames and passwords from your shared appSettings.json file, like so:

{
    "CHSAuthKey": "5E56F79E-E678-4BAF-8FCF-B50141FAD5C7",
    "Archer": {
        "url": "https://archer.company.com",
        "Production": {
            "instance": "Archer-Prod",
        },
        "UAT": {
            "instance": "Archer-UAT",
        }
    }
}

and put usernames and passwords in local user secrets, e.g.

{
    "Archer": {
        "username": "Developer1",
        "Production": {
            "password": "nbewaroija32"
        },
        "UAT": {
            "password": "*n2*)(h2jk)(__"
        }
    }
}

Then by using "configOverrideKey" parameter during the call to CoreConfig.Load() as follows:

var core = Estrelica.CoreConfig.Load(wm => Log(wm.Message), 
    configOverrideKey: "UAT");

this will cause Estrelica.Core to authenticate with

  • the "Archer-UAT" instance (defined by the "UAT" instance override label in appSettings.json)
  • located at https://archer.company.com (defined by the default "Archer" url in appSettings.json)
  • using the account name "Developer1" (defined by the default "Archer" username in the user secrets file)
  • and password "n2)(h2jk)(__" (defined by the password found in the "UAT" instance override settings in the user secrets file).

Even if you only have a single Archer instance, this override strategy can also be useful for testing your application under different user accounts with different roles or permissions in Archer, to see how those permissions affect your code execution. Just put all your Archer connection details in the default "Archer" section, then define override sections for each of the user accounts to be tested:

{
    "CHSAuthKey": "5E56F79E-E678-4BAF-8FCF-B50141FAD5C7",
    "Archer": {
        "url": "https://archer.company.com",
        "instance": "Archer-UAT",
        "SysadminTestScenario": {
            "username": "sysadmin",
            "password": "@rcher@dmin"
        },
        "GeneralUserTestScenario": {
            "username": "GeneralUser123",
            "password": "p@ssword321"
        }
    }
}

Then it's a simple matter of invoking CoreConfig.Load() alternately with configOverrideKey: "SysadminTestScenario" and configOverrideKey: "GeneralUserTestScenario" to see what differences arise in your code.