Quantcast
Channel: General .NET – Brian Pedersen's Sitecore and .NET Blog
Viewing all 167 articles
Browse latest View live

Read from Azure Queue with .NET Core

$
0
0

The documentation around Azure Queues and .NET Core is a little fuzzy, as the approach have changed slightly over the last updates. Previously you had a shared Storage Account NuGet Package that gave access to Queues, Blob Storage and Table storage. Now you use seperate NuGet packages, and the one for queues are:

The reading and writing to the queue have changed slightly also, so this is a simple but complete example on how to send and receive messages using Azure.Storage.Queues:

using Azure.Storage.Queues;
using Azure.Storage.Queues.Models;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace MyCode
{
  public class QueueRepository 
  {
    private int _batchCount;
    private QueueClient _client;

    /// <summary>
    /// Create a new instance of QueueRepository
    /// </summary>
    /// <param name="connectionString">Connectionstring to Azure Storage Account</param>
    /// <param name="queueName">Name of queue</param>
    /// <param name="batchCount">Number of messages to get from queue per call</param>
    public QueueRepository(string connectionString, string queueName, int batchCount)
    {
      _client = new QueueClient(connectionstring, queueName);
      _batchCount = batchCount;
    }

    /// <summary>
    /// Add a new message to the queue
    /// </summary>
    /// <param name="messageText">Contents of the message</param>
    public async Task Send(string messageText)
    {
	  // If you do not base64 encode the message before adding it, you
	  // cannot read the message using Visual Studio Cloud Explorer
      await _client.SendMessageAsync(Base64Encode(messageText));
    }

    /// <summary>
    /// Read a maximum of _batchCount messages from the queue. Once they are
	/// read, they are immediately deleted from the queue. This approach is
	/// not the default approach, and will negate the auto-retry machanism 
	/// built into the queue system. But it makes the queue easier to understand
    /// </summary>
    public async Task<IEnumerable<QueueMessage>> Receive()
    {
      int maxCount = _batchCount;
      int maxBatchCount = 32;
      List<QueueMessage> receivedMessages = new List<QueueMessage>();
      do
      {
        if (maxCount < 32)
          maxBatchCount = maxCount;
        QueueMessage[] messages = await _client.ReceiveMessagesAsync(maxBatchCount, TimeSpan.FromSeconds(30));
        receivedMessages.AddRange(messages);
        await DeleteMessageFromQueue(messages);

        if (messages.Count() < maxBatchCount)
          return receivedMessages;

        maxCount -= messages.Count();
      } while (maxCount > 0);

      return receivedMessages;
    }

    private async Task DeleteMessageFromQueue(QueueMessage[] messages)
    {
      foreach (QueueMessage message in messages)
      {
        await _client.DeleteMessageAsync(message.MessageId, message.PopReceipt);
      }
    }

    private static string Base64Encode(string plainText)
    {
      var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
      return System.Convert.ToBase64String(plainTextBytes);
    }
  }
}

EXPLANATION:

The constructor takes the azure storage account connection string and the queue name as parameters, along with a batchcount. The batchcount is how many messages are received from the queue per read.

The Send method will base64 encode the message before adding it to the queue. If you do not do this, you cannot read the message using Visual Studio Cloud Explorer.

The Receive method will read batchcount messages from the queue. Messages are deleted immediately after reading. If you do not do this, you will have to manually delete the message before 30 seconds, or the message will appear in the queue again. Delete the code that deletes from the queue if you wish that functionality.

Also, please note that messages that you receive are also base64 encoded. To decode the message you can use this QueueMessage Extension Method. The extension method will also convert a JSON queue message into an object.

MORE TO READ:


Read and Write blob file from Microsoft Azure Storage with .NET Core

$
0
0

The documentation on the Azure Storage Blobs are a little fuzzy, as the NuGet packages and the approach have changed over time.

The latest NuGet Package is now called:

The concept of blob storages are the same though:

  • You use a connectionstring to connect to an Azure Storage Account.
  • Blob storage is divided into containers. To access a container you need a BlobContainerClient.
  • To access a blob you get a BlobClient from a BlobContainerClient.
  • With the BlobClient you can upload and download blobs.

Blobs can be accessed via an URL, like this:

ENOUGH TALK, SHOW ME THE CODE:

This is an example of a very simple repository that will read, write and delete blobs:

using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using System.IO;
using System.Text;
using System.Threading.Tasks;

namespace MyCode
{
  public class BlobRepository
  {
    private BlobContainerClient _client;

    /// <summary>
    /// Create an instance of blob repository
    /// </summary>
    /// <param name="connectionString">The storage account connection string</param>
    /// <param name="containerName">The name of the container</param>
    public BlobRepository(string connectionString, string containerName)
    {
      _client = new BlobContainerClient(connectionString, containerName);
      // Only create the container if it does not exist
      _client.CreateIfNotExists(PublicAccessType.BlobContainer);
    }

    /// <summary>
    /// Upload a local file to the blob container
    /// </summary>
    /// <param name="localFilePath">Full path to the local file</param>
    /// <param name="pathAndFileName">Full path to the container file</param>
    /// <param name="contentType">The content type of the file being created in the container</param>
    public async Task Upload(string localFilePath, string pathAndFileName, string contentType)
    {
      BlobClient blobClient = _client.GetBlobClient(pathAndFileName);

      using FileStream uploadFileStream = File.OpenRead(localFilePath);
      await blobClient.UploadAsync(uploadFileStream, new BlobHttpHeaders { ContentType = contentType });
      uploadFileStream.Close();
    }

    /// <summary>
    /// Download file as a string
    /// </summary>
    /// <param name="pathAndFileName">Full path to the container file</param>
    /// <returns>Contents of file as a string</returns>
    public async Task<string> Download(string pathAndFileName)
    {
      BlobClient blobClient = _client.GetBlobClient(pathAndFileName);
      if (await blobClient.ExistsAsync())
      {
        BlobDownloadInfo download = await blobClient.DownloadAsync();
        byte[] result = new byte[download.ContentLength];
        await download.Content.ReadAsync(result, 0, (int)download.ContentLength);

        return Encoding.UTF8.GetString(result);
      }
      return string.Empty;
    }

    /// <summary>
    /// Delete file in container
    /// </summary>
    /// <param name="pathAndFileName">Full path to the container file</param>
    /// <returns>True if file was deleted</returns>
    public async Task<bool> Delete(string pathAndFileName)
    {
      BlobClient blobClient = _client.GetBlobClient(pathAndFileName);
      return await blobClient.DeleteIfExistsAsync(DeleteSnapshotsOption.IncludeSnapshots);
    }
  }
}

HOW TO USE IT?

// Create a new repository. Enter the connectionstring and the
// container name in the constructor:
BlobRepository rep = new BlobRepository("YOUR_SECRET_CONNECTIONSTRING","test");

// To upload a file, give a file name from the local disk.
// Add the name of the blob file (notice that the path is in the blob filename
// not, the container name), and the content type of the file to be uploaded
await rep.Upload("d:\\test.json", "blobtest/test.json", "application/json")

// To download a file, enter the path and file name of the blob
// (not including the container name) and the contents is returned as a 
// string
string jsonFile = await rep.Download("blobtest/test.json");

// Deleting the blob is done by entering the path and file of the blob
await rep.Delete("blobtest/test.json");

In the example above, the URL to the created blob will be:

This is just a simple example, but it can easily be modified to uploading streams or downloading binary files.

That’s it. Happy coding.

MORE TO READ:

C# Remove Duplicates from List with LINQ

$
0
0

C# LINQ do have a Distinct() method that works on simple types:

// An string with non-unique elements
string s = "a|b|c|d|a";

// Split the list and take the distinctive elements
var distinctList = s.Split('|').Distinct().ToArray();

// Re-join the list 
var distinctString = string.Join("|", distinctList);

// Output will be: "a|b|c|d"
Console.WriteLine(distinctString);

For non-simple types, you have 2 options, but first lets make a non-simple type, a class:

public class MyClass
{
    public string Title { get; set; }
    public string Text { get; set; }
}

OPTION 1: IMPLEMENT AN EQUALITYCOMPARER

The equalitycomparer is a class that is specifically designed to compare a specific class. An example that will compare on the Title property of the MyClass looks like this:

public class MyClassDistinctComparer : IEqualityComparer<MyClass>
{
    public bool Equals(MyClass x, MyClass y) 
    {
        return x.Title == y.Title;
    }

    public int GetHashCode(MyClass obj) 
    {
        return obj.Title.GetHashCode() ^ obj.Text.GetHashCode();
    }       
}

And to use it:

// Create a list of non-unique titles
List<MyClass> list = new List<MyClass>();
list.Add(new MyClass() { Title = "A", Text = "Text" });
list.Add(new MyClass() { Title = "B", Text = "Text" });
list.Add(new MyClass() { Title = "A", Text = "Text" });

// Get the distinct elements:
var distinctList = list.Distinct(new MyClassDistinctComparer());

// Output is: "A B"
foreach (var myClass in distinctList)
     Console.WriteLine(myClass.Title);

OPTION 2: GROUP AND TAKE FIRST

If an equalitycomparer is too much of a hassle, you can take a shortcut and group the list by the title, and take the first element in each group:

// Make a list of non-unique elements
List<MyClass> list = new List<MyClass>();
list.Add(new MyClass() { Title = "A", Text = "Text" });
list.Add(new MyClass() { Title = "B", Text = "Text" });
list.Add(new MyClass() { Title = "A", Text = "Text" });

// Skip the equalitycomparer. Instead, group by title, and take the first element of each group
var distinctList = list.GroupBy(s => s.Title).Select(s => s.First()).ToArray();

// Output is: "A B"
foreach (var myClass in distinctList)
    Console.WriteLine(myClass.Title);

What happens is, that the GroupBy(s => s.Title) will make 2 groups, one for title “A” with 2 elements, and one for title “B” with 1 element. The Select(s => s.First()) then takes the first element from each group, resulting in a list with unique elements.

MORE TO READ:

HttpClient follow 302 redirects with .NET Core

$
0
0

The HttpClient in .NET Core will not automatically follow a 302 (or 301) redirect. You need to specify that you allow this. use the HttpClientHandler to do this:

private static HttpClient _httpClient = new HttpClient(
    new HttpClientHandler 
    { 
        AllowAutoRedirect = true, 
        MaxAutomaticRedirections = 2 
    }
);

Now your code will follow up to 2 redirections. Please note that a redirect from a HTTPS to HTTP is not allowed.

You can now grab the contents even from a redirected URL:

public static async Task<string> GetRss()
{
    // The /rss returns a 301 Moved Permanently, but my code will redirect to 
    // /feed and return the contents
    var response = await _httpClient.GetAsync("https://briancaos.wordpress.com/rss");
    if (!response.IsSuccessStatusCode)
        throw new Exception($"The requested feed returned an error: {response.StatusCode}");

    var content = await response.Content.ReadAsStringAsync();
    return content;
}

MORE TO READ:

Sitecore read from Custom Database

$
0
0

It is very easy to read from a custom database in your Sitecore code.

STEP 1: ADD THE CONNECTIONSTRING TO CONNECTIONSTRINGS.CONFIG

This is an example of a .config file that transforms the connection strings for existing Sitecore databases (like the core, master, web etc.) and includes a connection string to a custom database.

<connectionStrings xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <add name="core" connectionString="Data Source=xxx;Initial Catalog=xxx_Core;User ID=xxx;Password=xxx" xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
  <add name="master" connectionString="Data Source=xxx;Initial Catalog=xxx_Master;User ID=xxx;Password=xxx" xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
  <add name="web" connectionString="Data Source=xxx;Initial Catalog=xxx_Web;User ID=xxx;Password=xxx" xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
  ...
  ...
  <add name="CustomDatabase" connectionString="Data Source=xxx;Initial Catalog=xxx_CustomDatabase;User ID=xxx;Password=xxx" providerName="System.Data.SqlClient" xdt:Locator="Match(name)" xdt:Transform="InsertIfMissing"/>
  ...
  ...
</connectionStrings>

STEP 2: GET THE CONNECTIONSTRING FROM Sitecore.Configuration.Settings:

Reading the connection string is easy:

private static readonly string _connectionString =
    Sitecore.Configuration.Settings.GetConnectionString("CustomDatabase");

STEP 3: DO YOUR MAGIC:

This is an example of a class that uses Dapper to read data from a custom database:

using Dapper;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;

namespace MyCode
{
  internal static class MyRepository
  {
    private static readonly string _connectionString = Sitecore.Configuration.Settings.GetConnectionString("CustomDatabase");

    internal static IEnumerable<CustomData> Get(int customKey)
    {
      using (var conn = new SqlConnection(_connectionString))
      {
        var trail = conn.Query<CustomData>(Constants.StoredProcedures.Get, new { CustomKey = customKey }, commandType: CommandType.StoredProcedure);
        return trail;
      }
    }
  }
}

MORE TO READ:

Sitecore publishItem pipeline – handling missing delete when publishing

$
0
0

When publishing items in Sitecore, the publishItem pipeline is called for each item that must be published, or unpublished. Except for children of items that have been deleted. Let’s take this example:

Items To Publish
Items to publish

If I publish the top item, the publishItem pipeline will be called for every child with the PublishAction.PublishVersion property in the context.

If I then delete “Page1” and republish the top item, the publishItem pipeline is run again. This time the “Page1” will be called with PublishAction.DeleteTargetItem property in the context. But there is no publishItem being called for each of the children of “Page1“.

This is because Sitecore assumes that when a parent is being unpublished, it will automatically clean up any child items, because the relationship between parent and child have been broken.

SO THAT IS THE CHALLENGE THEN?

In my solution, we synchronize certain Sitecore items with an external database. This database have no parent-child relation. so when the “Page1” is deleted, the children below is not deleted from the external database, because no one tells the database to do so.

HOW CAN THIS BE FIXED:

My colleague Odin Jespersen helped me fix this issue. He created a new pipeline step for the publishItem pipeline that compares the web database with the master database to find items that have been deleted from the master database but not in the web database. These items are deemed deleteable and we then delete the items from the external database.

ENOUGH TALK, SHOW ME THE CODE:

This is how it looks like:

STEP 1: ADD THE STEP TO THE PUBLISHITEM PIPELINE:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:env="http://www.sitecore.net/xmlconfig/env/">
  <sitecore>
    <pipelines>
      <publishItem>
        <processor patch:before="*[@type='Sitecore.Publishing.Pipelines.PublishItem.DetermineAction, Sitecore.Kernel']" type="MyCode.DeleteFromExternalDatabase, MyDll" />
      </publishItem>
    </pipelines>
  </sitecore>
</configuration>

STEP 2: THE PROCESSOR:

using System.Linq;
using Sitecore;
using Sitecore.Data.Items;
using Sitecore.Globalization;
using Sitecore.Publishing;
using Sitecore.Publishing.Pipelines.PublishItem;

namespace MyCode
{
  public class DeleteFromExternalDatabase : PublishItemProcessor
  {
    private readonly MasterDatabaseFactory _sourceDatabaseFactory = Sitecore.Data.Database.GetDatabase("master");
    private readonly WebDatabaseFactory _targetDatabaseFactory = Sitecore.Data.Database.GetDatabase("web");

    public override void Process([CanBeNull] PublishItemContext context)
    {
      if (context == null)
        return;
      if (context.Aborted)
        return;
      if (context?.PublishContext == null)
        return;
      
      Item sourceItem = context.PublishHelper.GetSourceItem(context.ItemId);
      if (sourceItem == null)
        return;
      
      if (!sourceItem.Paths.IsContentItem)
        return;
      
	  // Only do this trick for the items that are being synchronized with the 
	  // external database
	  if (sourceItem.TemplateID != MyTemplate)
        return;
      
	  Item targetFolder = _targetDatabaseFactory.Get().GetItem(sourceItem.ID);
      if (targetFolder == null)
        return;
      
      var targetItems = targetFolder.GetChildrenDerivedFrom(MyTemplate);
      if (targetItems != null)
      {
        foreach (Item targetItem in targetItems)
        {
          if (_sourceDatabaseFactory.Get().GetItem(targetItem.ID) != null)
          {
            continue;
          }
          var items = targetItem.GetChildrenDerivedFrom(MyTemplate);
          if (items == null)
            continue;
          foreach (Item item in items)
          {
            if (_sourceDatabaseFactory.Get().GetItem(item.ID) == null)
            {
              // Do the delete from the external database
            }
          }
        }
      }
    }
  }
}

MORE TO READ:

Write to SOLR from .NET Core

$
0
0

.NET Core supports the SOLR index through the SolrNet.Core NuGet packages.

Here is how you update the index.

STEP 1: THE NUGET PACKAGES

You need the following NuGet packages:

STEP 2: CREATE A MODEL CLASS CONTAINING THE PROPERTIES OF YOUR SOLR INDEX

Use the SolrNet.Attributes to specify the properties of the document to index:

using System;
using SolrNet.Attributes;

namespace MyCode
{
  public class MySolrDocument
  {
    [SolrField("_uniqueid")]
    [SolrUniqueKey]
    public int RowID { get; set; }

    [SolrField("name")]
    public string Name { get; set; }

    [SolrField("age")]
    public int Age { get; set; }
  }
}

STEP 3: INJECT SOLR INTO YOUR SERVICECOLLECTION

Your code needs to know the Solr URL and which model to return when the Solr instance is queried. This is an example on how to inject Solr, your method might differ slightly:

private IServiceProvider InitializeServiceCollection()
{
    ...
    ...
    .AddSolrNet<MySolrDocument>("https://myurl:8983/solr/myindex")
    .AddSingleton<MySolrRepository>()
    .BuildServiceProvider();
    return services;
}

STEP 4: IMPLEMENT A REPOSITORY THAT CAN UPDATE THE INDEX

This is a simple repository that can update the SOLR index:

using System;
using System.Collections.Generic;
using SolrNet;
using SolrNet.Commands.Parameters;
using System.Linq;
using System.Threading.Tasks;

namespace MyCode
{
  public class MySolrRepository
  {
    private readonly ISolrOperations<MySolrDocument> _solr;

    public MappedValueRepository(ISolrOperations<MySolrDocument> solr)
    {
      _solr = solr;
    }

    public async Task Update(int id, string name, int age)
    {
        MySolrDocument solrDoc = new MySolrDocument()
        {
            RowId = id,
            Name = name,
            Age = age
        };
        await _solr.AddAsync(solrDoc);
        await _solr.CommitAsync();
    }

  }
}

Explanation:

SOLR does not have separate create and update statements. If a document with the same unique key exists, the document is updated. If not, it is created.

SOLR updates are protected using transactions. You need to call CommitAsync() before your changes are committed. SOLR also provides a RollbackAsync() method in case you need to roll back.

To delete documents, simply call DeleteAsync() with a model class containing the unique key specified in the model class.

That’s it. You are now a SOLR expert.

MORE TO READ:

C# Working with Namespaces in XDocument and XML

$
0
0

XDocument is your old XML friend in C#, and it allows you to create or manipulate XML documents. To work with namespaces, you must use the XNamespace class.

This is an imaginary XML document (a Google Merchant Feed formatted one) that we wish to manipulate:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:g="http://base.google.com/ns/1.0">
	<channel>
		<item>
			<g:id>42069</g:id>
			<title>Brian Caos</title>
			<description>A developer</description>
			<link>https://briancaos.wordpress.com/</link>
			<g:image_link>https://briancaos.files.wordpress.com/2019/09/bp-small-e1567426409278.png</g:image_link>
			<g:availability>in stock</g:availability>
			<g:price>42.00 USD</g:price>
		</item>
	</channel>
</rss>

Notice that the XML have the namespace http://base.google.com/ns/1.0, but not all fields are namespaced. This is because the document is basically a RSS 2.0 feed with additional fields, and it’s only the additional fields that have the namespace.

READ THE DOCUMENT

To read the document from a string, you need an XDocument, and a XNamespace:

// Get the RSS 2.0 XML data
string feedData = {{feed string}}
 
// Convert the data into an XDocument
var document = XDocument.Parse(feedData);
// Specify the Google namespace
XNamespace g = "http://base.google.com/ns/1.0";

The {{feed string}} is of course the feed as described above.

The trick is that we now have the namespace in the variable called “g”. To add a namespace, you simply add the “g” variable in front of the element name, like this:

ADD AN ELEMENT WITH A NAMESPACE

If I would like to add a <g:availability>in_stock</g:availability> to every item in the XML document, I can do so like this:

// Get all elements with the name "item"
var items = document.Descendants().Where(node => node.Name == "item");
foreach (var item in items)
{
    // Create new element with namespace http://base.google.com/ns/1.0
    var id = new XElement(g + "availability", "in stock");
    // Add it to the document
    item.Add(id);
}

GET AN ELEMENT WITH A NAMESPACE

This will get all items with the element is <g:availability>out of stock</g:availability>:

items = document.Descendants()
  .Where(node => node.Name == "item"
         && node.Descendants()
         .Any(desc => desc.Name == g + "availability"
              && desc.Value == "out of stock"));

MORE TO READ:


Run parallel tasks in batches using .NET Core, C# and Async coding

$
0
0

If you have several tasks that you need to run, but each task takes up resources, it can be a good idea to run the tasks in batches. There are a few tools out there, and one of them is the SemaphoreSlim class. The SemaphoreSlim limits the number of concurrent threads with a few easy lines of code.

But first we need an imaginary method that we need to run in parallel:

private async Task DoStuff(string value)
{
  // pseudo code, you just need to imagine
  // that this method executes a task
  await something(value);
}

And we need a bunch of DoStuff values that need to run:

List<string> values = new List<string>();
values.Add("value1");
values.Add("value2");
values.Add("value3");
...
...
...

Now, to start “DoStuff()” with all the values from the “values” list, but run them in batches, we can do this:

using System.Threading;

public async Task BatchRun(List<string> values)
{
    // Limit the concurrent number of 
    // threads to 5
    var throttler = new SemaphoreSlim(5);

    // Create a list of tasks to run
    var tasks = values.Select(async value =>
    {
        // ... but wait for each 5 tasks
        // before running the next 5 tasks
        await throttler.WaitAsync();
        try
        {
            // Run the task
            return await DoStuff(value);
        }
        catch (Exception exception)
        {
            // handle the exception if any
        }
        finally
        {
            // Always release the semaphore
            // when done
            throttler.Release();
        }
    });

    // Now we actually run the tasks
    await Task.WhenAll(tasks);
}

The code will first setup a SemaphoreSlim and ask it to throttle the number of threads to 5. Next it will generate a list of tasks, and use the WaitAsync() to ask the SemaphoreSlim to wait for the first 5 tasks to finish before starting the next 5 tasks. Finally, the Task.WhenAll() method is running the tasks.

The greatness of the SemaphoreSlim lies within the simplicity of the code. You control the batch count one place, and all you need to do is to call WaitAsync() and Release() as extra lines of code. The rest looks like any other task runner.

MORE TO READ:

C# and Microsoft.ML. Removing words from sentence. Getting started with Machine Learning

$
0
0

Microsoft.ML is the NuGet package for ML.Net, Microsoft’s open-source Machine Learning framework.

In this introduction I will create a stopword engine, capable of removing unwanted words from a sentence. I know that it is overkill to use machine learning to do this, but it serves as a great introduction as how to initialize and call Microsoft.ML.

STEP 1: THE NUGET PACKAGE

You need the following NuGet package:

STEP 2: CREATE A LIST OF STOPWORDS

We need a list or unwanted words to remove from the list:

public class StopWords
{
  internal static readonly string[] Custom =
  {
    "profanity",
    "swearing",
    "degrading"
  };
}

STEP 3: CREATE THE TEXTPROCESSING SERVICE

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Transforms.Text;

public class TextProcessingService
{
  // The PredictionEngineis part of the Microsoft.ML package
  private readonly PredictionEngine<InputData, OutputData> _stopWordEngine;

  // The PredictionEngine receives an array of words
  private class InputData
  {
    public string[] Words { get; set; }
  }

  // The PredictionEngine returns an array of words
  private class OutputData : InputData
  {
    public string[] WordsWithoutStopWords { get; set; }
  }

  public TextProcessingService()
  {
    var context = new MLContext();

    // Getting the list of words to remove from our sentece
	var stopWords =
      StopWords.Custom.ToArray();

    // Define the transformation
	var transformerChain = context.Transforms.Text
      .RemoveDefaultStopWords(
        inputColumnName: "Words",
        outputColumnName: "WordsWithoutDefaultStopWords",
        language: StopWordsRemovingEstimator.Language.English)
      .Append(context.Transforms.Text.RemoveStopWords(
        inputColumnName: "WordsWithoutDefaultStopWords",
        outputColumnName: "WordsWithoutStopWords",
        stopwords: stopWords));

    var emptySamples = new List<InputData>();
    var emptyDataView = context.Data.LoadFromEnumerable(emptySamples);
    var textTransformer = transformerChain.Fit(emptyDataView);

    _stopWordEngine = context.Model.CreatePredictionEngine<InputData, OutputData>(textTransformer);
  }

  public string[] ExtractWords(string text)
  {
      // This will remove stopwords
	  var withoutStopWords = _stopWordEngine.Predict(new InputData { Words = text.Split(' ')}).WordsWithoutStopWords;
      if (withoutStopWords == null)
        return null;
      return withoutStopWords;
  }
}

USAGE:

public static void Main()
{
  var textProcessing = new TextProcessingService();
  var newString = textProcessing.ExtractWords("my code removes swearing and degrading language");
  Console.WriteLine(String.Join(' ',newString));
}

The code above will generate the following output:

  • code removes language

But why does it do that? The input string is “my code removes swearing and degrading language” and I have only defined “swearing” and “degrading” as words that needs to be removed?

The answer lies within line 37 in the TextProcessingService. I use a StopWordsRemovingEstimator, and the language is set to English. The RemoveDefaultStopWords method will add these default stop words to my list of words. The Microsoft class is pre-loaded with a number of stopwords, among those “my“, “and“. My list of words just adds to that list.

That’s it. Happy coding.

MORE TO READ:

Sitecore poor Index Update performance linked to missing Index in the Links database

$
0
0

Suddenly my index updates took forever to finish. I mean, one index update would take 2 minutes. And I have a lot of index updates. A lot.

UPDATE: 2022-01-19: Sitecore have suggested the same changes to the links database in KB1000639 – How to reduce query execution time for the Link Database.

Running jobs in Sitecore
Index Updates in Sitecore

After some panic and some SQL debugging we were lead to some long running SQL statements that looked like this:

SELECT *
  FROM [Sitecore_Web].[dbo].[Links]
  where TargetItemID = '562F77DD-6C00-4BE1-AF0E-9F9EEAA8CCEF' 
  and TargetDatabase = 'master'

One of these would take almost 2 minutes to finish.

Now, my master database contains 2.000.000+ items, increasing with at least 5.000 items per week. Each of my items points to a category item, and each category item can have 250.000+ links to them. So there is a lot of rows in the Links database. 36.000.000 to be exact.

Inside the Links database there is several indexes, but none of them on TargetItemID and TargetDatabase. So we tried to create the index ourselves:

CREATE NONCLUSTERED INDEX [IX_TargetDatabaseTargetItemID] ON [dbo].[Links]
(
	[TargetDatabase] ASC,
	[TargetItemID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO

And voila! After a system restart, my index updates was back on track.

FINAL NOTES:

First of all, I usually don’t create indexes on databases that I do not own. And I cannot guarantee that the index solved all of my problems. It just so happens that after the index was introduced, my system is running fine.

MORE TO READ:

Sitecore create custom Content Editor Warnings using Conditions and Rules

$
0
0

The Content Editor Warnings are these yellow boxes that hover over your content in the Content Editor in Sitecore:

Content Editor Warning
Content Editor Warning

In the old days we would make them by hooking into the getContentEditorWarnings pipeline. Nowadays we use the Sitecore Rules engine. So to make your own Content Editor Warning you need to:

  • Create a custom Condition.
  • Create the code for the custom Condition
  • Create a new Rule

STEP 1: CREATE A CUSTOM CONDITION

Custom Condition
Custom Condition

Conditions are placed under:

/sitecore/system/Settings/Rules/Definitions/Elements

You can create your own folder in this structure and create a “Condition“. The “Text” field should contain the text displayed in the Rule editor, and the “Type” field should contain the reference to the class containing the code, for example:

MyCode.Rules.MyRule, MyDll

STEP 2: CREATE THE CODE

A condition inherits from a RuleContext. Rules can return different things, my return true or false, but you can create rules that return values like integers or strings.

This is my Rule:

using Sitecore.Diagnostics;
using Sitecore.Data.Items;
using Sitecore.Rules;
using Sitecore.Rules.Conditions;

namespace MyCode.Rules
{
  public class MyRule<T> : WhenCondition<T> where T : RuleContext   
  {
    protected override bool Execute(T ruleContext)
    {
      Assert.ArgumentNotNull(ruleContext, "ruleContext");
      // This is them item being clicked on
	  Item item = ruleContext.Item;
	  // Pseudocode, imagine that you are checking
	  // something
	  if (something == true)
        return true;
      return false;
    }
  }
}

STEP 3: CREATE A NEW RULE

Content Editor Warning Rule
Content Editor Warning Rule

Content Editor Warning rules are placed in:

/sitecore/system/Settings/Rules/Content Editor Warnings/Rules

And you need to create a new Content Editor Warning Rule.

Then you create the rule using the rules editor:

Rules Editor
Rules Editor

My rule is fancy, as a first check for a specific template type (built in rule), then check for my own rule.

Lastly, the rule ends with what should happen. Here you type in the text that should appear in the content editor warning.

That’s it. You are now a Sitecore expert.

MORE TO READ:

C# Lists in Lists – Getting all inner values, or the unique inner values

$
0
0

Getting the inner value of a list inside a list can seem complicated in C#, but LINQ makes it easier. But first let’s make a list in a class, and then make a list of the class with the list of classes – you get the drift:

using System;
using System.Collections.Generic;
using System.Linq;

public class OuterClass 
{
	public IEnumerable<InnerClass> InnerClasses { get; set; }
}

public class InnerClass
{
	public string InnerValue { get; set; }	
}

Then let’s make a list of the OuterClass, and assign some values:

// The starting point: A list of OuterClasses
List<OuterClass> outerClasses = new List<OuterClass>();

// Make a list of InnerClasses, and add 3 values, A,B,C
List<InnerClass> innerClasses = new List<InnerClass>();
innerClasses.Add(new InnerClass() { InnerValue = "A" });
innerClasses.Add(new InnerClass() { InnerValue = "B" });
innerClasses.Add(new InnerClass() { InnerValue = "C" });

// Add a OuterClasses, each with a list of 3 InnerClasses
outerClasses.Add(new OuterClass() { InnerClasses = innerClasses });
outerClasses.Add(new OuterClass() { InnerClasses = innerClasses });
outerClasses.Add(new OuterClass() { InnerClasses = innerC

Now we have a list of 3 OuterClasses, and each OuterClass have a list of 3 InnerClasses, a total of 9 items, and a list of InnerValue like this: A,B,C,A,B,C,A,B,C

THE NON-LINQ WAY:

Without LINQ, you should make 2 loops:

foreach (var outerClass in outerClasses)
{
	foreach (var innerClass in innerClasses)
	{
		Console.WriteLine(innerClass.InnerValue);	
	}
}

This returns A,B,C,A,B,C,A,B,C

THE LINQ WAY:

Use the SelectMany to get all InnerClasses:

var allInnerClasses = outerClasses.SelectMany(o => o.InnerClasses);
foreach (var innerClass in allInnerClasses)
{
	Console.WriteLine(innerClass.InnerValue);	
}

This also returns A,B,C,A,B,C,A,B,C

GETTING ONLY THE UNIQUE VALUES:

To return only the unique values, do this:

var allUniqueInnerValues = outerClasses.SelectMany(o => o.InnerClasses)
	.GroupBy(i => i.InnerValue)
	.Select(i => i.First())
	.Select(i => i.InnerValue);
	
foreach (var innerValue in allUniqueInnerValues)
{
	Console.WriteLine(innerValue);
}

This will return only A,B,C.

MORE TO READ:

Writing to Azure EventHub using EventHubBufferedProducerClient

$
0
0

The Azure Event Hub is a big data queue system capable of handling millions of events per second and packs nifty tricks in both the read and write end. I have written plenty of articles on how to read and write from the Azure Queue, and the Azure Queue is often the better choice for your basic queue needs, but this time we are rolling out the big guns.

BASIC EVENT HUB INFO:

  • The Event Hub likes to receive data in batches. This lowers the connections created for each write. The EventDataBatch class was invented for this.
  • Partitioning is a thing, especially for the read end. Storing your data with the right partition data can increase performance considerably.

Now, the EventHubProducerClient is the basic client for writing to the Event Hub, and it will handle writing in batches. But each batch can only write to one partition.

To overcome the partitioning problem, Microsoft have the EventHubBufferedProducerClient Class. It’s currently only available in the Azure.Messaging.EventHubs v5.7.0-beta.2 package, but I assume it will survive the beta stage.

The difference between EventHubProducerClient  and EventHubBufferedProducerClient is that in the latter you just add an event to an internal queue, and the class will do the writing in the background at intervals. This allows you to batch-drop events to the Event Queue, and the client will then handle the partitioning and the writing for you.

But enough talk, let’s code:

STEP 1: THE NUGET PACKAGE

You need to allow for beta packages, as you (currently) need this package:

STEP 2: THE CODE:

using System.Threading.Tasks;
using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;

namespace MyCode
{
  public class EventHubRepository
  {
    private static EventHubBufferedProducerClient _producerClient;

    public EventHubRepository()
    {
      _producerClient = new EventHubBufferedProducerClient("connectionstring", "event hub name");
      _producerClient.SendEventBatchFailedAsync += args =>
      {
        // Do something if an error happens
        return Task.CompletedTask;
      };
      _producerClient.SendEventBatchSucceededAsync += args =>
      {
        // Do something if all goes right. You can 
        // log the number of writes per partition for example:
        // Console.Log(args.PartitionId);
        // Console.Log(args.EventBatch.Count);
        return Task.CompletedTask;
      };
    }

    public async Task AddMessage(string partitionKey, string message)
    {
      var enqueueOptions = new EnqueueEventOptions { PartitionKey = partitionKey };
      await _producerClient.EnqueueEventAsync(new EventData(message), enqueueOptions);
    }
  }
}

Nothice how there is no implicit write method? The class will flush the events to the Event Hub when it’s time to do so. If you have 8 partitions, the class will do 8 batch writes. You can control the time using the EventHubBufferedProducerClientOptions class.

WHAT IS THE PARTITION KEY?

The partition key keeps related events together in the same partition and in the exact order in which they arrive. The partition key is some string that defines a relation between your data. Let’s say you are storing page views per sessionid. The order of views per session is important, so you use the sessionid as partition key to ensure that all page views from the same session is stored in the same partition. This ensures their sort order, and makes it faster to process page views for one session in the other end.

That’s it. You are now a Azure Event Hub expert.

MORE TO READ:

C# DateTime to UNIX timestamps

$
0
0

Although the UNIX timestamp (also known as the EPOCH time or the POSIX time) probably does not make sense to you, it is still widely used. After all, who cares how many seconds have elapsed since Jan 1st 1970, right? Well, lets not worry about that, instead lets see how we can convert a standard C# DateTime into a UNIX timestamp.

The trick is to convert the DateTime into a DateTimeOffset, and convert it from there to the UNIX timestamp number.

Now, there is 2 versions of the UNIX timestamp, one just displaying the number of seconds and another one where the milliseconds are added:

// Get the offset from current time in UTC time
DateTimeOffset dto = new DateTimeOffset(DateTime.UtcNow);
// Get the unix timestamp in seconds
string unixTime = dto.ToUnixTimeSeconds().ToString();
// Get the unix timestamp in seconds, and add the milliseconds
string unixTimeMilliSeconds = dto.ToUnixTimeMilliseconds().ToString();

The code above will have the following results:

  • Current UTC time: 2/24/2022 10:37:13 AM
  • UNIX timestamp: 1645699033
  • UNIX timestamp, including milliseconds: 1645699033012

If you wish, you can wrap this in an Extension method on your DateTime class:

public static class DateTimeExtensions
{
	// Convert datetime to UNIX time
	public static string ToUnixTime(this DateTime dateTime)
	{
		DateTimeOffset dto = new DateTimeOffset(DateTime.UtcNow);
		return dto.ToUnixTimeSeconds().ToString();
	}

	// Convert datetime to UNIX time including miliseconds
	public static string ToUnixTimeMilliSeconds(this DateTime dateTime)
	{
		DateTimeOffset dto = new DateTimeOffset(DateTime.UtcNow);
		return dto.ToUnixTimeMilliseconds().ToString();
	}
}

MORE TO READ:


.NET 6.0 and Simple (minimal) Api’s: Create an Api without controllers

$
0
0

Have you always dreamt of skipping the controllers in your Web Api? Are you the lazy type? And have you always wanted to return data directly from your repositories?

Feat not, .NET 6.0 to your rescue. The new “Minimal Api” feature have you covered. No more controller boilerplate code just to return a simple value. Fewer lines of code means less code to maintain. And it’s probably easier to find all the endpoints as they can be found one place.

STEP 1: CREATE AN ASP.NET CORE WEB API

You must create an ASP.NET Core Web Api, and select .NET 6.0 as your platform.

See here how to create a Web Api Project.

STEP 2: CREATE A REPOSITORY

This silly mock repository will return a string.

namespace SimpleApi.Repositories
{
  public class MyRepository
  {
    public async Task<string> HelloWorldAsync()
    {
      // This is a stupid example, but just an example.
      // You would of course have a real async method here
      return await Task.Run(() => "Hello world");
    }

    public string HelloWorldNoAsync()
    {
      // Just an example to prove that you don't need
      // async methods. It's just that it is best practice to
      // do so, it's not required.
      return "Hello World";
    }

    public async Task<string> HelloAsync(string who)
    {
      // This is a stupid example, but just an example.
      // You would of course have a real async method here
      return await Task.Run(() => $"Hello {who}");
    }
  }
}

STEP 3: ADD ENDPOINTS TO Program.cs

using SimpleApi.Repositories;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

// Add my own repository to the service collection
builder.Services.AddSingleton<MyRepository>(new MyRepository());

// Build the app
var app = builder.Build();

// Add Swagger when in development mode
if (app.Environment.IsDevelopment())
{
  app.UseSwagger();
  app.UseSwaggerUI();
}

// And always use https
app.UseHttpsRedirection();

// Add 1st enpoint: An example without async
app.MapGet("/api/helloworld-noasync", (MyRepository myRep) => myRep.HelloWorldNoAsync()).WithName("Hello World Without Async");
// Add 2nd endpoint: An example with async
app.MapGet("/api/helloworld-async", async (MyRepository myRep) => await myRep.HelloWorldAsync()).WithName("Hello World With Async");
// Add 3rd endpoint: An example with parameter and async
app.MapGet("/api/hello/{who}", async (string who, MyRepository myRep) => await myRep.HelloAsync(who)).WithName("Hello Who With Async");

// Start the api website
app.Run();

SO WHAT’S HAPPENING HERE?

Well, In.NET 6.0 you no longer need a public static void Main(string[] args), not a namespace. Not that it’s not there, it’s just implicitly added by the compiler.

The first lines of code (3-24) will do the plumbing, add Swagger and set up https.

Line 27-29-31 adds the endpoints. This is the interesting part. These are the 3 endpoints that my API will expose. I have made 3 examples.

The app.MapGet() will create GET methods on the implied path. You can use app.MapPost(), app.MapPut() and many more to create POST, PUT or any method you wish.

  • The first endpoint is how to call a method without async. This is generally not recommended, async is the preferred method.
  • The second endpoint is how to call an async method. This will just return “Hello World” but serves as an example on how to call an async method,
  • The last endpoint shows how to add parameters to the endpoint. This will return “Hello {who}” where {who} is the parameter in the endpoint.

That’s it. You are now a Minimal API expert. Happy coding.

MORE TO READ:

C# Deserialize JSON to dynamic ExpandoObject()

$
0
0

The Microsoft System.Text.Json.Serialization.JsonSerializer cannot (yet) convert JSON into dynamic objects. But good ole Newtonsoft.Json can.

The conversion is simple too:

// This is my json string
string s = "
{
    "label": "MyLabel",
    "values": {
      "Key1": "Value1",
      "Key2": "Value2",
      "Key2": "Value3",
    }
  }";
  
// This is the conversion
dynamic json = 
  JsonConvert.DeserializeObject<ExpandoObject>(
    s, 
    new ExpandoObjectConverter()
  );

After the conversion, you can work with the dynamic object as you wish:

string label = json.label;
foreach (var value in json.values)
  Console.WriteLine($"{value.Key}: {value.Value}");

MORE TO READ:

Read and Update Azure App Configuration settings with C#

$
0
0

Azure App Configuration is a nifty way of storing configuration settings in the cloud, and with the live reload feature you can change the config settings and they will automatically update in your app without a need for redeploy.

But the configuration explorer is not the best tool in the world. It’s simple and does not give a very good overview of your settings. Especially if you use the labels feature.

This simple interface is not good. Especially with labels enabled.

But fear not. If you are an automation geek or just want to update the settings using code, this is – of course – possible.

STEP 1: THE NUGET PACKAGE

You will need this NuGet package:

STEP 2: CONNECT TO AZURE APP CONFIGURATION

// Connect to an App Configuration Client using a connectionstring
private static readonly string _connectionString = "Endpoint=https://mtaz-appconfig.azconfig.io...=";
private readonly ConfigurationClient _client = new ConfigurationClient(_connectionString);

STEP 3: READ ALL SETTINGS USING A FILTER

Filters use * notation to identity wildcards. In this example all my settings are named MySetting:[something], for example MySetting:FixCasing or MySetting:HideInSearch:

var settingsSelector = new SettingSelector() { KeyFilter = "MySetting:*" };
var settings = _client.GetConfigurationSettings(settingsSelector);

If you use labels, you can first sort by label, then get all settings:

public dynamic GetAllSettings()
{
  var settingsSelector = new SettingSelector() { KeyFilter = "MySetting:*" };
  var settings = _client.GetConfigurationSettings(settingsSelector);
  var labels = settings.GroupBy(s => s.Label);

  List<dynamic> list = new List<dynamic>();
  foreach (var label in labels)
  {
    dynamic s = new ExpandoObject();
    s.label = label.Key != null ? label.Key : string.Empty;
    var setingsByLabel = settings.Where(s => s.Label == label.Key);
    var values = new Dictionary<string, object>();
    foreach (var setting in setingsByLabel)
    {
      values.Add(setting.Key, setting.Value);
    }
    s.values = values;
    list.Add(s);
  }
  return list;
}

When serialized, the above returns the following output:

[
  {
    "label": "Development",
    "values": {
      "MySetting:ConvertCategoryToGroup": "false",
      "MySetting:FieldToGetCategoryFrom": "title",
      "MySetting:FixCasing": "false"
    }
  },
  {
    "label": "Production",
    "values": {
      "MySetting:ConvertCategoryToGroup": "false",
      "MySetting:FieldToGetCategoryFrom": "description",
      "MySetting:FixCasing": "false"
    }
  }
]

STEP 4: ADD, UPDATE AND DELETE SETTINGS

This is equally easy:

// Will add a setting it the setting is not already there
public void AddConfiguration(string key, string, value, string label)
{
  _client.AddConfigurationSetting(value.Key, value.Value, label);
}

// Will add a setting it the setting is not already there, 
// or update the setting if it already exist  
public void UpdateConfiguration(string key, string, value, string label)
{
  _client.SetConfigurationSetting(value.Key, value.Value, label);
}

// Deletes a single configuration setting
public void DeleteConfiguration(string key, string label)
{
  _client.DeleteConfigurationSetting(setting.Key, label);
}

// Deletes all settings matching my filter from a labeled section	
public void DeleteEntireConfigurationSection(string label)
{
  var settingsSelector = new SettingSelector() { KeyFilter = "MySetting:*", LabelFilter = label };
  var settings = _client.GetConfigurationSettings(settingsSelector);
  foreach (var setting in settings)
    _client.DeleteConfigurationSetting(setting.Key, label);
}

The DeleteEntireConfigurationSection is especially nifty, as you can wipe all settings from an entire labeled section.

That’s it. You are now an Azure App Configuration Expert. Happy coding.

MORE TO READ:

C# .NET Core API Versioning with Microsoft.AspNetCore.Mvc.Versioning

$
0
0

.NET Core allows you to control versions of your APIs. To do so you need the following NuGet package:

But lets try to version our API.

STEP 1: CREATE A CONTROLLER

using Microsoft.AspNetCore.Mvc;
using System;

namespace Controllers.V1
{
  [Route("v{version:apiVersion}/[controller]")]
  public class HelloWorldController : Controller
  {
    [HttpGet]
    public async Task<IActionResult> HelloWorld()
    {
      // This is a stupid example, but just an example.
      // You would of course have a real async method here
      return await Task.Run(() => Ok("Hello world"));
    }
  }
}

Notice how I in line 6 defines the route as v{version:apiVersion}/[controller]? The {version:apiVersion} will define the versioning that we use later on.

STEP 2: ADD VERSIONING TO THE SERVICES

In your startup code, add the versioning to the services:

using Microsoft.AspNetCore.Mvc.Versioning;
using Microsoft.AspNetCore.Mvc;

...
...

services.AddApiVersioning(
  options =>
  {
    options.ReportApiVersions = true;
    options.Conventions.Controller<Controllers.V1.HelloWorldController>().HasApiVersion(new ApiVersion(1, 0));
  });

Here we have now defined that the Controllers.V1.HelloWorldController have version 1.

STEP 3: CALL YOUR ENDPOINT

The endpoint will now respond to the following URL:

  • /v1/helloworld

WHY NOT JUST HARDCODING THE API VERSION TO THE CONTROLLER ROUTE?

Versioning have several advantages. This is not all it can do:

You can deprecate versions by adding the following attribute to the controller:

[Route("v{version:apiVersion}/[controller]")]
[ApiVersion("1.0", Deprecated = true)]
public class HelloWorldController : Controller

You can map a specific method in a controller to a specific version by adding the following attribute to the method. So one controller can have several versions of an endpoint:

[HttpGet]
[MapToApiVersion("1.0")]
public async Task<IActionResult> HelloWorld()

And you can do a whole lot of other version related stuff. But this will get you started.

Please note that versioning is not supported when using .NET 6 Miminal API’s, but there is a plan to implement it later.

You are now an API versioning expert. Happy coding.

MORE TO READ:

Host .net core 6.0 WebApplication as Kestrel Windows Service

$
0
0

If you create a .NET Core 6.0 Web Application as default, the application will run as an App Service just fine. But if you wish to host the application yourself, you can run the application as a Windows Service.

This is how you usually initialize the Web Application in .NET Core 6:

var builder = WebApplication.CreateBuilder(args);

To implement Windows Service capabilities, you need the following Nuget package:

Microsoft.Extensions.Hosting.WindowsServices

And you need to change the WebApplication.CreateBuilder:

var webApplicationOptions = new WebApplicationOptions() { 
  Args = args,
  ContentRootPath = AppContext.BaseDirectory, 
  ApplicationName = System.Diagnostics.Process.GetCurrentProcess().ProcessName 
};
var builder = WebApplication.CreateBuilder(webApplicationOptions);
builder.Host.UseWindowsService();

Windows services are rooted in the System32 folder, which is why the WebApplicationOptions changes the content root path back to the base directory of the application itself.

That’s it. You are now a .NET Core 6.0 expert. Happy coding.

MORE TO READ:

Viewing all 167 articles
Browse latest View live