Trending client performance - Lab 03

Sergey Sorokin — Wed, 19 May 2010 21:18:06 GMT

If you are curious about the number of Trend controls you can run against your CSWorks server infrastructure, you may find this post interesting. We will make minor changes to historical data server configuration, run a few Trend control clients against it and analyze what we see.

Environment

CSWorks 1.2.3800.0
Server: Intel Core 2 Duo @ 2.40GHz, 2 GB RAM, Windows Server 2008
Client: Intel Core 2 Duo T5300 @ 1.73GHz (notebook), 2 GB RAM, Windows XP SP3
Network: Wireless 54 Mbps

Server Configuration

1. As usual, we have to configure History Recorder so it uses some scalable database. Install SQL Server 2008 Express on your server machine.

2. Create database "CSWorks"

3. Create HistoricalData table - see "createCommand" parameter of <dbtarget ...="" name="Standard SQLServer DbTarget"> in CSWorks.Server.HistoryRecorderService.exe.config.

4. Configure SQL Server data source and make it active in CSWorks.Server.HistoryRecorderService.exe.config:

<dbTargetConfig>
  <dbTargets activeDbTarget="Standard SQLServer DbTarget">
    <dbTarget name="Standard SQLite DbTarget" ...
      ...
    />
    <dbTarget name="Standard SQLServer DbTarget"
      providerInvariantName="System.Data.SqlClient"
      connectionString="Data Source=localhost\sqlexpress; Initial Catalog=CSWorks;user id=sa;password=...;"
      ...
    />
  </dbTargets>
</dbTargetConfig>

5. Configure HistoryReaderWebService to read historical data from this database. In the web.config, assign SQLServer target to the primary partition and specify correct connection string:

<historyTopology>
    <historyPartitions>
      <historyPartition name="partition1" primaryDbTarget="partition1 Primary DbTarget (SQLServer)" secondaryDbTarget="">
        ..
      </historyPartition>
    </historyPartitions>
</historyTopology>

<dbTargetConfig>
    <dbTargets>
      ...
      <dbTarget name="partition1 Primary DbTarget (SQLServer)"
                providerInvariantName="System.Data.SqlClient"
                connectionString="Data Source=localhost\sqlexpress; Initial Catalog=CSWorks;user id=sa;password=...;"
                ...
                />
    </dbTargets>
</dbTargetConfig>

6. Restart HistoryRecorder service and verify that it writes observation to the newly configured database.

Running clients

Before running the clients, make sure you have prepared *.clientConfig in CSWorks.Client.TrendDemo.xap to run from a remote machine. Please see this post for details.

Now run a few Trend clients on your client machine using this command:

start iexplore "http://myserver/CSWorksDemo/TrendDemo.html"

I ran 25 instances, increasing the load by 5-instance chunks.

Results

All 25 clients run without problems, trending data (both live and historical) arrives without delays, server seems to be perfoming fine. Here is a screenshot made on the server machine. Clients consume about 200K of live and historical data every second, server machine uses about 35% of its CPU capacity. SQL Server and ASP.NET worker process are working hard (14% and 18%, respectively) to deliver historical data to the clients.

The spikes in data transfer rates mark moments when Trend control were re-querying bigger amounts of historical data - View->Tracking setting were set to On for all Trend control instances, and all instances were refreshing the whole picture synchronously (well, in chunks of 5 instances, of course).

The spikes in CPU consumption mark moments when I ran chunks of 5 IE instances.

Summary

Not bad for a commodity server box that runs every piece of the deployment: all CSWorks services (LiveData, Alarm, HistoryRecorder), web services and database engine. Potential bottlenecks are:
- the database - not surprising;
- web service layer - but we can scale it out using web farm.

History Recorder performance - Lab 02

Sergey Sorokin — Tue, 20 Apr 2010 17:09:46 GMT

In the latest release of CSWorks, we have improved Historical Data Server performance. Let's see what History Recorder is capable of now.

Environment

CSWorks 1.2.3730.0
Server: Intel Core 2 Quad Q6600 @ 2.40GHz, 4 GB RAM, Windows 7

Set-up

1. Install SQL Server 2008 Express on your server machine.

2. Create database "CSWorks"

3. Create HistoricalData table - see "createCommand" parameter of in CSWorks.Server.HistoryRecorderService.exe.config.

4. Configure SQL Server data source and make it active in CSWorks.Server.HistoryRecorderService.exe.config:

Please note we tell History Recorder to keep alarm event records in the database for 10 minutes only, and we perform record cleaning every 30 seconds. This only because our SQL Server is not capable of taking heavy load.

5. Restart History Recorder service. Make sure events are now written to CSWorks database (run "select * from HistoricalData" to confirm it).

6. Using cscript tool, run a script that generates 2000 historical data points:

7. Copy generated historical data point descriptions to RecorderDataPoints.xml:

History Recorder will pick up the change in a couple of seconds and will start saving observation for those 2000 data points to the database. Give our setup some time to stabilize.

Results

After 10 minutes, History Recorder maintains about 3.5 million observation records in the database and writes about 6000 observation records every second on average. Database file size is between 1 and 1.5 gigabytes. Here is a screenshot with Performance Monitor and DbgView windows:

Since our test live data changes in a very predictable way, there is a clear pattern in observation recording on the top perfmon chart. History Recorder memory consumption is under control too. Tracing shows that History Recorder deletes between 120000 and 270000 "obsolete" observations every 30 seconds. As you may have noticed, the maximum number of record it is allowed to delete in one shot is 300K, see 'maintenanceCommand' parameter above. Our setup is properly balanced, so History Recorder does not reach this limit.

If you add more historical datapoints and make total count, say 5000, you may end up in a situation when History Recorder simply cannot write all collected observations in a timely manner, and they will accumulate in the memory buffer. Major symptom will be growing memory consumption by History Recorder. CSWorks 1.2.3800.0 introduces "Write Buffer Size" performance counter that shows current number observations to be written to the database by HistoryRecorder, so this overload scenario becomes more obvious.

Summary

Please plan your historical data management carefully. Use scalable database engine, and give it a lot of spare CPU resources. Use multiple History Recorder machines if needed. If the amount of data is extremely big, use multiple databases and apply partitioning technique described in CSWorks documentation.

ControlSystemWorks Blog - historical data

Trending client performance - Lab 03

History Recorder performance - Lab 02