If you are curious how well our new BACnet IP implementation can perform, you may find this post interesting. I used  CSWorks 2.0.4115.0 LiveData Service installed on a Core2 Quad Q6600 @2.40 GHz machine with 4GB RAM running 64-bit Windows 7. As a testing client, I used four instances of a simple LiveData client application that subscribes to updates from 5000 BACnet analog inputs, requesting for updates every second. Test applications talks to CSWorks LiveData Service directly over WCF (exctly the same way WCF LiveData Agent demo does). All hardware is connected via 100mb Ethernet.

If you are curious about the number of Trend controls you can run against your CSWorks server infrastructure, you may find this post interesting. We will make minor changes to historical data server configuration, run a few Trend control clients against it and analyze what we see.

Environment

CSWorks 1.2.3800.0
Server: Intel Core 2 Duo @ 2.40GHz, 2 GB RAM, Windows Server 2008
Client: Intel Core 2 Duo T5300 @ 1.73GHz (notebook), 2 GB RAM, Windows XP SP3
Network: Wireless 54 Mbps

Server Configuration

1. As usual, we have to configure History Recorder so it uses some scalable database. Install SQL Server 2008 Express on your server machine.

2. Create database "CSWorks"

3. Create HistoricalData table - see "createCommand" parameter of <dbtarget ...="" name="Standard SQLServer DbTarget"> in CSWorks.Server.HistoryRecorderService.exe.config.

4. Configure SQL Server data source and make it active in CSWorks.Server.HistoryRecorderService.exe.config:

<dbTargetConfig>
  <dbTargets activeDbTarget="Standard SQLServer DbTarget">
    <dbTarget name="Standard SQLite DbTarget" ...
      ...
    />
    <dbTarget name="Standard SQLServer DbTarget"
      providerInvariantName="System.Data.SqlClient"
      connectionString="Data Source=localhost\sqlexpress; Initial Catalog=CSWorks;user id=sa;password=...;"
      ...
    />
  </dbTargets>
</dbTargetConfig>

5. Configure HistoryReaderWebService to read historical data from this database. In the web.config, assign SQLServer target to the primary partition and specify correct connection string:

  <historyTopology>
    <historyPartitions>
      <historyPartition name="partition1" primaryDbTarget="partition1 Primary DbTarget (SQLServer)" secondaryDbTarget="">
        ..
      </historyPartition>
    </historyPartitions>
  </historyTopology>

  <dbTargetConfig>
    <dbTargets>
      ...
      <dbTarget name="partition1 Primary DbTarget (SQLServer)"
                providerInvariantName="System.Data.SqlClient"
                connectionString="Data Source=localhost\sqlexpress; Initial Catalog=CSWorks;user id=sa;password=...;"
                ...
                />
    </dbTargets>
  </dbTargetConfig>

6. Restart HistoryRecorder service and verify that it writes observation to the newly configured database.

Running clients

Before running the clients, make sure you have prepared *.clientConfig in CSWorks.Client.TrendDemo.xap to run from a remote machine. Please see this post for details.

Now run a few Trend clients on your client machine using this command:

start iexplore "http://myserver/CSWorksDemo/TrendDemo.html"

I ran 25 instances, increasing the load by 5-instance chunks.

Results

All 25 clients run without problems, trending data (both live and historical) arrives without delays, server seems to be perfoming fine. Here is a screenshot made on the server machine. Clients consume about 200K of live and historical data every second, server machine uses about 35% of its CPU capacity. SQL Server and ASP.NET worker process are working hard (14% and 18%, respectively) to deliver historical data to the clients.

The spikes in data transfer rates mark moments when Trend control were re-querying bigger amounts of historical data - View->Tracking setting were set to On for all Trend control instances, and all instances were refreshing the whole picture synchronously (well, in chunks of 5 instances, of course).

The spikes in CPU consumption mark moments when I ran chunks of 5 IE instances.

Summary

Not bad for a commodity server box that runs every piece of the deployment: all CSWorks services (LiveData, Alarm, HistoryRecorder), web services and database engine. Potential bottlenecks are:
- the database - not surprising;
- web service layer - but we can scale it out using web farm.

In the latest release of CSWorks, we have improved Historical Data Server performance. Let's see what History Recorder is capable of now.

Environment

CSWorks 1.2.3730.0
Server: Intel Core 2 Quad Q6600 @ 2.40GHz, 4 GB RAM, Windows 7

Set-up

1. Install SQL Server 2008 Express on your server machine.

2. Create database "CSWorks"

3. Create HistoricalData table - see "createCommand" parameter of in CSWorks.Server.HistoryRecorderService.exe.config.

4. Configure SQL Server data source and make it active in CSWorks.Server.HistoryRecorderService.exe.config:

<dbTargetConfig>
  <dbTargets activeDbTarget="Standard SQLServer DbTarget">
    <dbTarget name="Standard SQLite DbTarget" ...
      ...
    />
    <dbTarget name="Standard SQLServer DbTarget"
      providerInvariantName="System.Data.SqlClient"
      connectionString="Data Source=localhost\sqlexpress; Initial Catalog=CSWorks;user id=sa;password=...;"
      createCommand="..."
      writeCommand="..."
      maintenanceCommand="delete top (300000) from HistoricalData ..."
      maxObservationAge="600"
      writeInterval="1"
      maintenanceInterval="30"
      maxQueryLength="65535"
      queryDelimiter=";"
      writeTxnBeginCommand="..."
      writeTxnCommitCommand="..."
    />
  </dbTargets>
</dbTargetConfig>

Please note we tell History Recorder to keep alarm event records in the database for 10 minutes only, and we perform record cleaning every 30 seconds. This only because our SQL Server is not capable of taking heavy load.

5. Restart History Recorder service. Make sure events are now written to CSWorks database (run "select * from HistoricalData" to confirm it).

6. Using cscript tool, run a script that generates 2000 historical data points:

function main()
{
  var areas = 200;
  var dpsInArea = 50;

  for (i = 0; i < areas; i++)
  {
    var areaId = i.toString();
    while(areaId.length < 4)
    {
      areaId = "0" + areaId;
    }

    WScript.Echo("<!-- area 0000" + areaId + "-AAAA-0000-0000-000000000000 -->");

    for (j = 0; j < dpsInArea; j++)
    {
      var dpId = j.toString();
      while(dpId.length < 4)
      {
        dpId = "0" + dpId;
      }
      dpId = areaId + dpId;
      WScript.Echo("<dataPoint id='{00000000-0000-0000-0000-0000" + dpId + "}' description='Tank fill - " + dpId + "' expression='tank1-" + dpId +"'/>");
    }
  }
}

main();

7. Copy generated historical data point descriptions to RecorderDataPoints.xml:

<dataPoints>
  ...
  <dataPoint id='{00000000-0000-0000-0000-000000000000}' description='Tank 1 fill - 0000' expression='tank1-0000'/>
  ...
  <dataPoint id='{00000000-0000-0000-0000-000000001999}' description='Tank 1 fill - 1999' expression='tank1-1999'/>
</dataPoints>

History Recorder will pick up the change in a couple of seconds and will start saving observation for those 2000 data points to the database. Give our setup some time to stabilize.

Results

After 10 minutes, History Recorder maintains about 3.5 million observation records in the database and writes about 6000 observation records every second on average. Database file size is between 1 and 1.5 gigabytes. Here is a screenshot with Performance Monitor and DbgView windows:

Since our test live data changes in a very predictable way, there is a clear pattern in observation recording on the top perfmon chart. History Recorder memory consumption is under control too. Tracing shows that History Recorder deletes between 120000 and 270000 "obsolete" observations every 30 seconds. As you may have noticed, the maximum number of record it is allowed to delete in one shot is 300K, see 'maintenanceCommand' parameter above. Our setup is properly balanced, so History Recorder does not reach this limit.

If you add more historical datapoints and make total count, say 5000, you may end up in a situation when History Recorder simply cannot write all collected observations in a timely manner, and they will accumulate in the memory buffer. Major symptom will be growing memory consumption by History Recorder. CSWorks 1.2.3800.0 introduces "Write Buffer Size" performance counter that shows current number observations to be written to the database by HistoryRecorder, so this overload scenario becomes more obvious.

Summary

Please plan your historical data management carefully. Use scalable database engine, and give it a lot of spare CPU resources. Use multiple History Recorder machines if needed. If the amount of data is extremely big, use multiple databases and apply partitioning technique described in CSWorks documentation.

I am always curious about my system performance. In this lab I will give CSWorks Alarming subsystem a hard time and see how it can handle it.

Environment

CSWorks 1.1.3700.0
Server: Intel Core 2 Quad Q6600 @ 2.40GHz, 3.25 GB RAM, Windows XP SP3
Client: Intel Core 2 Duo T5300 @ 1.73GHz (notebook), 2 GB RAM, Windows XP SP3
Network: Wireless 54 Mbps

Set-up

1. Install SQL Server 2008 Express on your server machine.

2. Create database "CSWorks"

3. Create AlarmEvents table - see "createCommand" parameter of <dbTarget name="Standard SQLServer DbTarget".../> in CSWorks.Server.AlarmService.exe.config.

4. Configure SQL Server data source in CSWorks.Server.AlarmService.exe.config:

<dbTargets activeDbTarget="Standard SQLServer DbTarget">
  <dbTarget name="Standard SQLServer DbTarget"  ...  connectionString="Data Source=myserverbox\SQLEXPRESS; Initial Catalog=CSWorks;user id=sa;password=youknowit" ...  maxEventAge="3600" .../>

Please note we tell CSWorks Alarm Service to keep alarm event records in the database for an hour so we can accumulate some amount of load data.

5. Restart CSWorks Alarm Service. Make sure events are now written to CSWorks database (run "select * from AlarmEvents" to confirm it).

6. Configure Silverlight web services URL for Alarm Demo application, so it can run from remote machine - make sure that ServiceReferences.ClientConfig in

C:\Program Files\CSWorks\Demo\Web\ClientBin\CSWorks.Client.AlarmDemo.xap

says

<endpoint address="http://myserverbox/CSWorksDemo/AlarmWebService/Service.asmx" ...>

For more details, see this post. Starting from version 1.4.3820.0, there is no need to do that, see this post for details.

7. Run Alarm Demo from a remote client and make sure it works properly.

8. Create a script that generates 3000 alarm descriptions in 10 alarm groups (download it here),  run "cscript genalarms.js > alarms.txt", copy generated alarm groups to Alarms.xml, alarmGroups element. Here is an excerpt:

<alarmGroup id='00000000-AAAA-0000-0000-000000000000' description='Alarm group #0000'>
  <alarms>
    <alarm id='{00000000-0000-0000-0000-000000000000}' description='Total intake tank load, id 00000000' expression='tank1 + tank2 + tank3 + tank4' saveHistory='true' ... >
      <thresholds>
        <threshold ... message='Total intake tank load too high: {0:#,##0.00}, exceeds {1:#,##0.00}, id 00000000' ... />
        ...
    <alarm id='{00000000-0000-0000-0000-000000090299}' description='Total intake tank load, id 00090299' expression='tank1 + tank2 + tank3 + tank4' saveHistory='true' ... >
      <thresholds>
        <threshold ... message='Total intake tank load too high: {0:#,##0.00}, exceeds {1:#,##0.00}, id 00090299' ... />
        ...
      </thresholds>
    </alarm>
  </alarms>
</alarmGroup>

These generated alarms contain references to a couple of deployment variables: totalIntakeTankLoadMax and totalIntakeTankLoadMin. Add them to Alarms.xml as well:

  <deploymentStates>
    <deploymentState name="Normal" isActive="true">
      <deploymentVariables>
        ...
        <deploymentVariable name="totalIntakeTankLoadMax" type="Int32" value="380"/>
        <deploymentVariable name="totalIntakeTankLoadMin" type="Int32" value="250"/>
      </deploymentVariables>
    </deploymentState>
    <deploymentState name="Startup">
      <deploymentVariables>
        ...
        <deploymentVariable name="totalIntakeTankLoadMax" type="Int32" value="380"/>
        <deploymentVariable name="totalIntakeTankLoadMin" type="Int32" value="250"/>
      </deploymentVariables>
    </deploymentState>
    <deploymentState name="Shutdown">
      <deploymentVariables>
        ...
        <deploymentVariable name="totalIntakeTankLoadMax" type="Int32" value="380"/>
        <deploymentVariable name="totalIntakeTankLoadMin" type="Int32" value="250"/>
      </deploymentVariables>
    </deploymentState>
  </deploymentStates>

Save Alarms.xml and give CSWorks Alarm Service a couple of seconds to digest the changes (this piece doesn't scale of course, so only one server CPU is involved).

9. Add newly created alarm group ids to AlarmWebService web.config file:

  <alarmTopology>
    <alarmPartitions>
      <alarmPartition ...>
        <areas>

          <!-- Test groups -->

          <area id="{...}" name="Test groups">
            <alarmGroups>
               <alarmGroup alarmGroupId='00000000-AAAA-0000-0000-000000000000' description='Alarm group #0000'/>
              ...
              <alarmGroup alarmGroupId='00000009-AAAA-0000-0000-000000000000' description='Alarm group #0009'/>
            </alarmGroups>
          </area>
          ...

10. Subscribe to newly created alarm groups in Alarm Demo application, either using "Subscribe..." menu command:

Or you can do that in application XAML:

<alarm:AlarmSummary ... AlarmGroupIds="...;00000000-AAAA-0000-0000-000000000000;..." />

11. After running Alarm Demo on your client machine for a minute you will be able to observe 12000 new alarm events and considerable spikes in network usage:



Let It Brew

Go grab a coffee and have a chat with your colleagues while CSWorks Alarm Service generates some alarm events.

Analyze This

Let's look at the alarm activity: System Monitor -> CSWorksAlarmService performance object -> Alarm Events per second:



~730 alarm events per second on average with spikes up to 9000 events per second. This correlates with our setup, remember: an alarm becoming inactive is an event too, and we have 4 thresholds per alarm. So, each spike is basically a burst of 3000*6=18000 events generated in 5 seconds. Now let's look at the CPU usage on the server box:



- not bad, the load is distributed properly. Let's make sure that all those events actually ended up in the database:



- good, those 700 000 events definitely did. Now back to the client.

With filtering turned off, the client seems to handle the spikes pretty well: Alarm Summary items are updated with 1-3 seconds delays. Things are getting worse if you apply a filter (say, show only active alarms): delay bumps up to 15 seconds. And of course, one of the processors on the client box is overloaded by filtering and data presentation activities. And it's not easy to scale them out: Silverlight dispatcher is single-threaded. Ok, there is some space for improvement on the client side.

And a word about network traffic for the inquiring minds. Fiddler tool tells all the truth and it ain't that ugly:



- each 2.3 MB per spike translates into ~130 bytes per event. Keep in mind that every event carries a unique text message. Not bad.

Back to Normal

Restore Alarms.xml - you have made a copy of the original file before inserting 3000 alarm descriptions, didn't you?  Again, let CSWorks Alarm Service digest the changes and in a couple of seconds, alarm summary is almost clear, showing only a couple of events. The storm is over.

Summary

Server load is scaled properly, stable and predictable. Network traffic is predictable too, those performance freaks out there can gain some bandwidth using HTTP compression (yes, Silverlight supports it). Database performance doesn't seem to be an issue at all. The only bottleneck in this scenario is client CPU consumption when processing bursts of 18000 events. So, the recommendation is: if your operator needs to keep an eye on 18000 events fired in five seconds (which is very unikely), buy him/her a better client box.