Oqtane Tutorials
Oqtane Tutorials

Dynamic DataSources Tutorials

Tutorial Home › Dynamic DataSources
Data Sources Tutorial - use in C# / Razor Code
#1 Dynamic DataSources Tutorial
Use Dynamic DataSources in Visual Query
Requirements
Resources

Dynamic DataSources Tutorial

DataSources are the magic backbone of 2sxc. They can get data, filter it, sort it, and more. In many cases you will simply use the DataSources provided by 2sxc and the EAV system, but you can do even more. Specifically:

  • You can create your own Dynamic DataSources in C# and use them in your code and in Visual Query. This is a bit more advanced, but very powerful. Common scenarios are DataSources which do things or retrieve data which the built-in DataSources don't do.
  • You can create compiled DataSources in C# and use them in your code and in Visual Query. This is much more advanced, and not demonstrated in these tutorials, since it would be hard to install them.

Custom Dynamic DataSources - Introduction

Here we'll give you a first taste of Custom Dynamic DataSources - a new feature in 2sxc 15.

This allows you to create DataSources directly in your App, without compiling to DLL or anything.

This is a 10-line super-simple example of a DataSource called Basic101. It will only return a single item with the answer to the meaning of life 😉. Afterwards we'll explain in more detail what's happening.

⬇️ Result | Source ➡️

List of Data in the CSV DataSource (1)

  • Hello from Basic101 - the Answer: 42 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  // Create the Dynamic DataSource with the name "Basic101"
  var basic101 = Kit.Data.GetSource(name: "Basic101");
}
<h3>List of Data in the CSV DataSource (@basic101.List.Count())</h3>
<ul>
  @foreach (var item in AsList(basic101)) {
    <li>
      <strong>@item.Title</strong> - the Answer: @item.TheAnswer
    </li>
  }
</ul>

Source Code of Basic101.cs

public class Basic101 : Custom.DataSource.DataSource16
{
  public Basic101(MyServices services) : base(services)
  {
    ProvideOut(() => new {
      Title = "Hello from Basic101",
      TheAnswer = 42,
    });
  }
}

This is the same sample as before, but with a lot more explanations. Here's what's happening:

  1. Using Kit.Data.GetSource(name: "...") we retrieve the DataSource using the name
  2. The name references a file with the same name Basics101.cs located in the DataSources folder of the current App.
  3. The rest of the magic is explained in the source code of the DataSource - see below.

⬇️ Result | Source ➡️

List of Data in the CSV DataSource (1)

  • Hello from Basic101-Commented - the Answer: 42 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  // Create the Dynamic DataSource with the name "Basic101Commented"
  var basic101c = Kit.Data.GetSource(name: "Basic101Commented");
}
<h3>List of Data in the CSV DataSource (@basic101c.List.Count())</h3>
<ul>
  @foreach (var item in AsList(basic101c)) {
    <li>
      <strong>@item.Title</strong> - the Answer: @item.TheAnswer
    </li>
  }
</ul>

Source Code of Basic101Commented.cs

// 1.1 The class must have the same name as the file
// 1.2 It must inherit from Custom.DataSource.DataSource16 
public class Basic101Commented : Custom.DataSource.DataSource16
{
  // 2.1 The constructor must have the same name as the class
  // 2.2 It must also have a parameter of type MyServices
  // 2.3 It must call the base constructor with the same parameter
  // 2.4 This ensures that any internal functionality is initialized
  public Basic101Commented(MyServices services) : base(services)
  {
    // 3.1 The ProvideOut method must be called to define the output
    // 3.2 It must be called with a lambda expression (the () => part)
    // 3.3 In this example we're just returning a single object
    // 3.4 The object can be anonymous, as shown here
    ProvideOut(() => new {
      // Our object has a Title property with a hello-message
      // and another property with the answer to life, the universe and everything
      Title = "Hello from Basic101-Commented",
      TheAnswer = 42,
    });
  }
}

Custom Dynamic DataSources - Lists

The previous example just returned one item/entity. But of course you can also return a list of items. This is the more common case. Our sample basically will just return data it generated, but your code will usually get data from elsewhere and provide it as a list.

Return a list of items numbered 1...5 with random Guid identifier.

⬇️ Result | Source ➡️

List of Data in the DataSource (5)

  • Hello from ListBasic (1 / c28e7224-6408-4a36-9940-2d4bdd15a1a3) - Fav Number: 2742
  • Hello from ListBasic (2 / db2b010d-be87-477a-9a97-ad5055ab98e9) - Fav Number: 2742
  • Hello from ListBasic (3 / 097cd915-c36c-4de6-896d-582e072d32c7) - Fav Number: 2742
  • Hello from ListBasic (4 / 9f3d61c2-6b42-4d7e-931f-53394216c805) - Fav Number: 2742
  • Hello from ListBasic (5 / 541e1c46-918b-46be-a450-e1a46ce1f91d) - Fav Number: 2742 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  // Create the Dynamic DataSource with the name "Basic101"
  var basic101 = Kit.Data.GetSource(name: "ListBasic");
}
<h3>List of Data in the DataSource (@basic101.List.Count())</h3>
<ul>
  @foreach (var item in AsList(basic101)) {
    <li>
      <strong>@item.Title</strong> (@item.EntityId / @item.EntityGuid) - Fav Number: @item.FavoriteNumber
    </li>
  }
</ul>

Source Code of ListBasic.cs

// This sample uses some LINQ
using System.Linq;

public class ListBasic : Custom.DataSource.DataSource16
{
  public ListBasic(MyServices services) : base(services)
  {
    ProvideOut(() =>
      // For demo, create a few of these items using numbers 1 to 5
      Enumerable.Range(1, 5).Select(i => new {
        // Property with name "Id" is automatically used for the EntityId
        Id = i,
        // Property with name "Guid" is automatically used for the EntityGuid
        Guid = System.Guid.NewGuid(),
        Title = "Hello from ListBasic",
        FavoriteNumber = 2742,
      })
    );
  }
}

⬇️ Result | Source ➡️

Data in the DataSource (1)

  • Current Temperature: 1.4
  • WindSpeed: 15.1
  • WindDirection: 242
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  // Create the Dynamic DataSource with the name "DataFromWebService"
  var weather = Kit.Data.GetSource(name: "DataFromWebService");
}
<h3>Data in the DataSource (@weather.List.Count())</h3>
<ul>
  @foreach (var item in AsList(weather)) {
    <li>Current Temperature: @item.Temperature</li>
    <li>WindSpeed: @item.WindSpeed</li>
    <li>WindDirection: @item.WindDirection</li>
  }
</ul>

Source Code of DataFromWebservice.cs

using System.Net.Http;
using System.Linq;
using System.Text.Json.Serialization;   // For JsonPropertyName

public class DataFromWebService : Custom.DataSource.DataSource16
{
  public DataFromWebService(MyServices services) : base(services)
  {
    ProvideOut(() => {
      var response = new HttpClient()
        .GetAsync("https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&current_weather=true")
        .GetAwaiter()
        .GetResult();
      response.EnsureSuccessStatusCode();
      
      var responseBody = response.Content.ReadAsStringAsync()
        .GetAwaiter()
        .GetResult();
      var result = Kit.Convert.Json.To<WeatherData>(responseBody);

      return new {
        Temperature = result.Current.Temperature,
        WindSpeed = result.Current.WindSpeed,
        WindDirection = result.Current.WindDirection,
      };
    });
  }
}

// Helper classes for JSON deserialization
// Note that we're not using most of the properties, but we have them for completeness of the tutorial
public class WeatherData
{
  public double Latitude { get; set; }
  public double Longitude { get; set; }
  [JsonPropertyName("generationtime_ms")]
  public double GenerationTimeMs { get; set; }
  [JsonPropertyName("utc_offset_seconds")]
  public int UtcOffsetSeconds { get; set; }
  public string Timezone { get; set; }
  [JsonPropertyName("timezone_abbreviation")]
  public string TimezoneAbbreviation { get; set; }
  public double Elevation { get; set; }
  [JsonPropertyName("current_weather")]
  public CurrentWeather Current { get; set; }
}

public class CurrentWeather
{
  public double Temperature { get; set; }
  public double WindSpeed { get; set; }
  public double WindDirection { get; set; }
  public int WeatherCode { get; set; }
  public int Is_Day { get; set; }
  public string Time { get; set; }
}

Custom Dynamic DataSources - Configuration

Often you will need a DataSource that accepts some kind of parameters (configuration). The code must have [Configuration] attributes on the properties that should be configurable. When calling GetSource(...) use options: new { ... } to set the values.

This simple example will get a DataSource from a file, and pass some configuration options. Specifically we'll give it the AmountOfItems and FavoriteColor.

⬇️ Result | Source ➡️

List of Data in the WithConfig DataSource (3)

  • Hello from WithConfig #1 - Favorite Color: dark blue
  • Hello from WithConfig #2 - Favorite Color: dark blue
  • Hello from WithConfig #3 - Favorite Color: dark blue 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

 @{
    // Create the Dynamic DataSource with the name "WithConfig"
    // and set some configuration options
    var withConfig = Kit.Data.GetSource(name: "WithConfig", parameters: new {
      AmountOfItems = 3,
      FavoriteColor = "dark blue"
    });
  }
  <h3>List of Data in the WithConfig DataSource (@withConfig.List.Count())</h3>
  <ul>
    @foreach (var item in AsList(withConfig)) {
      <li>
        <strong>@item.Title</strong> - Favorite Color: @item.FavoriteColor
      </li>
    }
  </ul>

Source Code of WithConfig.cs

using System.Linq;
using ToSic.Eav.DataSource; // This namespace is for the [Configuration] attribute

public class WithConfig : Custom.DataSource.DataSource16
{
  public WithConfig(MyServices services) : base(services, "My.Magic")
  {
    ProvideOut(() => {
      var result = Enumerable.Range(1, AmountOfItems).Select(i => new {
        Title = "Hello from WithConfig #" + i,
        FavoriteColor,
      });
      return result;
    });
  }

  // This attribute [Configuration] creates a configuration "FavoriteColor"
  // In this example [Configuration] already knows about the fallback.
  // * The property getter calls Configuration.GetThis()
  // * GetThis() will automatically use the property name "FavoriteColor" to look up the config
  // * Calling an empty GetThis() will always return a string,
  //   so it's ideal for string-properties where the Fallback was specified before
  [Configuration(Fallback = "magenta")]
  public string FavoriteColor { get { return Configuration.GetThis(); } }

  // This attribute [Configuration] creates configuration "AmountOfItems"
  // * The property getter calls Configuration.GetThis()
  // * GetThis() will automatically use the property name "AmountOfItems" to look up the config
  //   and will use the default value of 1 if it was not specified
  [Configuration]
  public int AmountOfItems { get { return Configuration.GetThis(1); } }
}

Custom Dynamic DataSources - Process In-Data

DataSources be attached together. This means that you can create a DataSource which takes one or more DataSources as input.

  This example has a DataSource which receives data from an <em>upstream</em> source.     The upstream source is the list of all authors in this App.     Our DataSource will then filter this list, and only keep the authors with an odd ID.

⬇️ Result | Source ➡️

Data from the KeepOdd DataSource (2)

  • Terry (ID: 5339)
  • Ed (ID: 5349) 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  // Get the stream of all authors
  var authors = App.Data["Persons"];
  // Create the Dynamic DataSource "KeepOdd" and attach the authors to it
  var keepOdd = Kit.Data.GetSource(name: "KeepOdd", attach: authors);
}
<h3>Data from the KeepOdd DataSource (@keepOdd.List.Count())</h3>
<ul>
  @foreach (var item in AsList(keepOdd)) {
    <li>
      <strong>@item.EntityTitle</strong> (ID: @item.EntityId)
    </li>
  }
</ul>

Source Code of KeepOdd.cs

using System.Linq;

public class KeepOdd : Custom.DataSource.DataSource16
{
  public KeepOdd(MyServices services) : base(services)
  {
    ProvideOut(() => {
      // Make sure we have an In stream - otherwise return an error
      var inStream = TryGetIn();
      if (inStream == null) return Error.TryGetInFailed();

      // Return only odd items
      return inStream.Where(e => e.EntityId % 2 == 1);
    });
  }
}

If your DataSource requires attached data (In) you may get hard-to-debug errors. Because of this, the sample also has error handling for this. The following code uses the KeepOdd but forgets to attach the in.

⬇️ Result | Source ➡️

Data from the KeepOdd DataSource (1)

  • Error: Stream 'Default' not found (ID: 0)
    Message: This DataSource needs the stream 'Default' on the In to work, but it couldn't find it. 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  // Create the Dynamic DataSource "KeepOdd" and forget to attach the In
  // note: debug:false is only used to ensure developers don't see a real exception
  // this is just so the tutorial is easier to manage
  var keepOdd = Kit.Data.GetSource(name: "KeepOdd", debug: false);
}
<h3>Data from the KeepOdd DataSource (@keepOdd.List.Count())</h3>
<ul>
  @foreach (var item in AsList(keepOdd)) {
    <li>
      <strong>@item.EntityTitle</strong> (ID: @item.EntityId)
      <br>
      <strong>Message: </strong>
      @item.Message
    </li>
  }
</ul>

Source Code of KeepOdd.cs

using System.Linq;

public class KeepOdd : Custom.DataSource.DataSource16
{
  public KeepOdd(MyServices services) : base(services)
  {
    ProvideOut(() => {
      // Make sure we have an In stream - otherwise return an error
      var inStream = TryGetIn();
      if (inStream == null) return Error.TryGetInFailed();

      // Return only odd items
      return inStream.Where(e => e.EntityId % 2 == 1);
    });
  }
}
 Important: Note that the DataSource also has code to handle errors if the In was not attached.

Multiple Lists/Streams

DataSources can have one or more streams.

The default stream is called Default so you don't need to specify it. In the following example, we have a second stream called Settings.

⬇️ Result | Source ➡️

Data in the stream "Settings"

  • PageSize: 50
  • ShowStuff: True

List of Data in the Default stream (5)

  • Hello from ListMultiStream (1 / e148384b-29d9-4772-8882-3f85ec63b2dc) - Fav Number: 2742
  • Hello from ListMultiStream (2 / fd8c6fa4-fea4-4ec1-aba7-4df10fd4407f) - Fav Number: 2742
  • Hello from ListMultiStream (3 / 118f06b0-f0b8-47c2-96d9-acf0d24df967) - Fav Number: 2742
  • Hello from ListMultiStream (4 / 90dcfae4-61a8-4a71-b987-13c604cbe0a1) - Fav Number: 2742
  • Hello from ListMultiStream (5 / f259e2e7-de1f-463c-9dd1-2e0901e443a6) - Fav Number: 2742 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  // Create the Dynamic DataSource with the name "ListMultiStream"
  var multiStream = Kit.Data.GetSource(name: "ListMultiStream");
  var settings = AsDynamic(multiStream["Settings"].List.FirstOrDefault());
}
<h3>Data in the stream "Settings"</h3>
<ul>
  <li>PageSize: @settings.PageSize</li>
  <li>ShowStuff: @settings.ShowStuff</li>
</ul>
<h3>List of Data in the Default stream (@multiStream.List.Count())</h3>
<ul>
  @foreach (var item in AsList(multiStream)) {
    <li>
      <strong>@item.Title</strong> (@item.EntityId / @item.EntityGuid) - Fav Number: @item.FavoriteNumber
    </li>
  }
</ul>

Source Code of ListMultiStream.cs

// This sample uses Enumerable.Repeat, which needs this namespace
using System.Linq;

public class ListMultiStream : Custom.DataSource.DataSource16
{
  public ListMultiStream(MyServices services) : base(services)
  {
    // First stream "Settings" containing just one item
    ProvideOut(() => new {
      Title = "Settings Entity",
      PageSize = 50,
      ShowStuff = true,
    }, name: "Settings");

    // A second stream on the normal (unnamed) "Default"
    ProvideOut(() => Enumerable.Range(1, 5).Select(i => new {
        Id = i,
        guid = System.Guid.NewGuid(),
        Title = "Hello from ListMultiStream",
        FavoriteNumber = 2742,
      })
    );
  }
}

In some cases it makes sense to provide multiple streams which were processed at the same time. For example, when splitting data or when providing a stream Folders and Files which are still related.
The following sample takes the original Persons and splits them into odd/even streams in one preparation step.

⬇️ Result | Source ➡️

List of Data in the All stream (6)

  • Douglas (#5338)
  • Terry (#5339)
  • Neil (#5340)
  • George (#5344)
  • Raphael (#5348)
  • Ed (#5349)

List of Data in the Odd stream (2)

  • Terry (#5339)
  • Ed (#5349)

List of Data in the Even stream (4)

  • Douglas (#5338)
  • Neil (#5340)
  • George (#5344)
  • Raphael (#5348) 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources

@{
  var oddEven = Kit.Data.GetSource(name: "SplitOddEven", attach: App.Data["Persons"]);
  var all = oddEven.List;
  var odd = oddEven["Odd"].List;
  var even = oddEven["Even"].List;
}
<h3>List of Data in the All stream (@all.Count())</h3>
<ul>
  @foreach (var item in AsList(all)) {
    <li>
      <strong>@item.EntityTitle</strong> (#@item.EntityId)
    </li>
  }
</ul>
<h3>List of Data in the Odd stream (@odd.Count())</h3>
<ul>
  @foreach (var item in AsList(odd)) {
    <li>
      <strong>@item.EntityTitle</strong> (#@item.EntityId)
    </li>
  }
</ul>
<h3>List of Data in the Even stream (@even.Count())</h3>
<ul>
  @foreach (var item in AsList(even)) {
    <li>
      <strong>@item.EntityTitle</strong> (#@item.EntityId)
    </li>
  }
</ul>

Source Code of SplitOddEven.cs

using System.Linq;

public class SplitOddEven : Custom.DataSource.DataSource16
{
  public SplitOddEven(MyServices services) : base(services)
  {
    ProvideOut(() => TryGetIn());
    ProvideOut(() => Split().Odd, name: "Odd");
    ProvideOut(() => Split().Even, name: "Even");
  }

  private Cache Split() {
    // If already cached (eg. it's retrieving Even after Odd was already retrieved), return cache
    if (_cache != null)
      return _cache;

    // Make sure we have an In stream - otherwise return an error
    var inStream = TryGetIn();
    if (inStream == null) return new Cache { Odd = Error.TryGetInFailed(), Even = Error.TryGetInFailed() };

    // Build cache so we don't have to re-calculate when retrieving other streams
    return _cache = new Cache {
      Odd = inStream.Where(e => e.EntityId % 2 == 1),
      Even = inStream.Where(e => e.EntityId % 2 == 0)
    };
  }

  private Cache _cache;

  private class Cache {
    public object Odd;
    public object Even;
  }
}

Get Data from Other DataSources

DataSources can also get other DataSources. For example, a DataSource could get data from the App directly to process them.

The Authors DataSource demonstrates how to get all App data and then filter it using an inner EntityTypeFilter DataSource.

⬇️ Result | Source ➡️

Items in the DataSource (6)

  • Douglas
  • Terry
  • Neil
  • George
  • Raphael
  • Ed 
@inherits Custom.Hybrid.Razor14
@using ToSic.Razor.Blade
@using System.Linq
@using ToSic.Eav.DataSources 

@{
  // Create the Dynamic DataSource with the name "Basic101"
  var authors = Kit.Data.GetSource(name: "Authors");
}
<h3>Items in the DataSource (@authors.List.Count())</h3>
<ul>
  @foreach (var item in AsList(authors)) {
    <li>
      <strong>@item.EntityTitle</strong>
    </li>
  }
</ul>

Source Code of Authors.cs

using System.Linq;
using ToSic.Eav.DataSources;

public class Authors : Custom.DataSource.DataSource16
{
  public Authors(MyServices services) : base(services)
  {
    ProvideOut(() => {
      // Get the app data
      var appData = Kit.Data.GetAppSource();

      // Get the Content-Type Filter DataSource
      var contentTypeFilter = Kit.Data.GetSource<EntityTypeFilter>(attach: appData, parameters: new { TypeName = "Persons"});

      // Return all the items after filtering
      return contentTypeFilter.List;
    });
  }
}

 

 

Data Sources Tutorial - use in C# / Razor Code
#1 Dynamic DataSources Tutorial
Use Dynamic DataSources in Visual Query