Posting compressed data in JSON format to the ASP.NET website using WinForm

We have posted an article on how to post JSON data from WinForm to ASP.NET/ASHX in 2015 (as shown below). In the earlier article, it does not optimize the payload size and reduce the communication time.

    http://laucsharp.blogspot.my/2015/07/posting-data-in-json-format-to-aspnet.html

In order to speed up the communication between the WinForm client and ASP.NET/ASHX, we must reduce the payload size.To do that, the contents must be compressed before we upload the data and decompress upon receiving it.

There are 2 compression algorithms for HTTP/HTTPS communication: gzip and deflat.

Here is the code to compress the contents with GZip algorithm. You need this code in both WinForm and ASP.NET/ASHX.

        public static void GZip(Stream stream, string data)
        {
            byte[] b = System.Text.Encoding.UTF8.GetBytes(data);
            GZip(stream, b);
        }

        public static void GZip(Stream stream, byte[] data)
        {
            using (var zipStream = new GZipStream(stream, CompressionMode.Compress, true))
            {
                zipStream.Write(data, 0, data.Length);
            }
        }
To decompress the GZip contents. You need this code in both WinForm and ASP.NET/ASHX.

        public static string GUnZipToString(Stream stream)
        {
            byte[] b = GUnZip(stream);
            return System.Text.Encoding.UTF8.GetString(b);
        }

        public static byte[] GUnZip(Stream stream)
        {
            using (var zipStream = new GZipStream(stream, CompressionMode.Decompress, true))
            using (MemoryStream ms = new MemoryStream())
            {
                zipStream.CopyTo(ms);
                return ms.ToArray();
            }
        }
In the WinForm app, we used to write the JSON data directly to the HttpWebRequest like this:

        // write the json data into the request stream.
        using (StreamWriter writer = new StreamWriter(request.GetRequestStream()))
        {
                writer.Write(json_data);
        }

In order to reduce the payload, we should write the compressed data by doing this:

        // to indicate we accept gzip content.
        request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip");

        // to indicate that the content is compressed using gzip algorithm.
        request.Headers.Add("Content-Encoding", "gzip");

        GZip(request.GetRequestStream(), json_data);

 In the ASP.NET/ASHX, we used to read the contents like this:

        // get the contents from the request stream
        Stream stream = context.Request.InputStream;
        using (StreamReader r = new StreamReader(stream))
        {
            s = r.ReadToEnd();
        }

Now, we have to read the contents and then decompress it. "Content-Encoding" is the indicator whether the contents is in GZip, deflat or not compress.

        if (HelperFunc.IsGZipContent(context))
        {
                 // read the compressed content.
                s = GUnZipToString(context.Request.InputStream);          
        }
         else
        {
                // get the contents from the request stream
                Stream stream = context.Request.InputStream;
                using (StreamReader r = new StreamReader(stream))
                {
                        s = r.ReadToEnd();
                }
        }

Here is the function to check whether the contents is compressed in GZip format. Please take note that the sample codes in this article did not support "deflat" algorithm.

        public static bool IsGZipContent(System.Web.HttpContext ctx)
        {
            string enc = ctx.Request.Headers["Content-Encoding"];

            if (string.IsNullOrEmpty(enc))
            {
                return false;
            }

            return  enc.ToLower().Contains("gzip");
        }

You will find more information about the static and dynamic compression in IIS:

   https://docs.microsoft.com/en-us/iis/configuration/system.webserver/httpcompression/

To find out why we need to compress the data, please refers to the following link:

   https://developer.mozilla.org/en-US/docs/Web/HTTP/Compression

Queue processing design

Queue concept is straight forward - append new item at the tail and dequeue from the head. It does not tell you how to process the requests in the queue.

You can enqueue new requests with multiple threads and processing the requests with 1 worker thread or multiple threads. Then, here comes with the processing designs:

Sequential (synchronous) processing the requests

With only 1 worker thread processing the requests in the queue, this is going to be slower as compared to the multiple worker threads but it has advantages. The advantage of sequential processing is when it comes to file I/O operation or network I/O operation. Just imagine that the exit has one door and only one person is able to pass through at one time.

Parallel (asynchronous) processing the requests

Thanks to multi-core and multi-CPU. With more than 1 worker thread processing the requests, it reduces the client wait time (on the response) and increases the server throughput.

Multi-thread has nothing to do with the "reducing the processing time for a request"! The "processing time" is depending on your codes and the how much the codes can be optimized (in the development time and also run time).

If there is a long run request and the request is handling by a worker thread (in a thread-pool), then, that worker thread is blocked. It won't be able to handle other request until it finished the current request. And if you have many of long running requests, then, all threads in the thread pool might not be able to reduce the response time. The server is now consider as "busy" even though the client is able to keep sending their requests into the queue.

Can we have unlimited threads in the pool..?!

The answer is no. If you have instantiated too many threads without using thread pool, then, the memory will be depleted... and the program will crash. 

What happen to the requests in the queue if my program crash?

All requests will be disappear from the memory. Your program may be designed with auto-restart in case it crashed. But, you won't be able to recover the requests which was stored in the memory.

In case you need something better than storing the requests in memory..

1. Each thread in the thread pool must maintain individual processing statistics and these statistics will be updating to the database hourly. With this information, the system administrator will be able to tell how many worker threads were blocked, in which time zone and which request blocks it. Then, he will be able to justify whether to upgrade or replace the server.

The statistics to be keep track by each thread in the thread pool should include:

- Number of requests that has been processed
- Total processed time (ms)
- Highest processed time (ms)
- Request name for highest processed time

- Peak hour - 24 time zone in a day (just get the current hour value will do)

2. Using database as queue storage instead of memory - the main advantage is that the server will be able continue from where it's left before crashing. With this approach, there will be some overhead in storing the requests into database. But, those overhead is justifiable with the crash-proof design.

Another advantage of storing the requests into database is that the secondary server will be kick-in if all the worker threads in the thread pool (in the primary server) are blocked. The secondary server will be notify by the primary server (through WCF or socket). Then, the secondary server will query the database for the pending requests and process it accordingly.

Two ways to handle the respond to the client

- The result will be stored in the database and then notify the primary server. Then, the primary server will query the result and responds to the client.

- OR the result will be send to the primary server (through WCF or socket) and then responds to the client.

Errors that appear in the WCF client app...

I was setting a new server to host my WCF app and I hit the wall with the following error messages. I spent 2 days in solving this problem. I guess, many people is wondering how to resolve these interesting errors.

• The server has rejected the client credentials.
• The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.9687500'.
• The communication object, System.ServiceModel.ServiceHost, cannot be used for communication because it is in the Faulted state.

I have developed a WCF solution in WinForm which allows you to comment out certain settings in the server and client so that you can reproduce the above mentioned error message.

  https://github.com/lauhw/WcfWinForm

Please beware that if you are running the WCF server and client in the same computer, the above error might not appear until you run them in a separate computer.

In a nutshell:

1. Both server and client config file must have the same settings in the "Binding" record. This includes the "security mode".
2. The "identity" settings must be the same in both server and client config file.
3. In the live environment, make sure you remove the "MEX" endpoint unless you want anyone to be able to query the interface.
4. Don't forget to open the port in the firewall.

In case you are hitting the same error mentioned above, the fastest way to solve this problem is to run the test WCF solution (as mentioned above). Make sure that the test solution works. After that, compare the differences in app.config between the test solution and your solution.

Hope you don't have to spend 2 days to solve the configuration problem. ;)

Web Service (asmx) vs Handler (ashx)

In ASP.NET environment, it comes with quite extensive way of handling the web request. There is Page class (aspx), Handler (ashx), Web service (asmx), Web service in WCF (asmx), etc. It's all upto the developer to decide the best way to write their program.

Let's compare the Web Service (asmx) and Handler (ashx):

• With Web Service you can run your Visual Studio and add the web service as web reference. This will create the necessary classes for you. With web service, the serialization and deserialization of the data object will be handle by .Net framework. This simply means that you can work with Web Service without having deep knowledge in the "background" process.
• With Handler you must find out what is the parameter to be pass to the handler. No code generator that is able to help out over here. It provides very basic features to serve the request and you will have to serialize/deserialize your data objects manually (you may use JavaScriptSerializer to deal with the JSON data). It does not limit you from returning the data in JSON, XML or CSV format. It's all depend on your design.

With Web Service, it saves you from lots of codes in the serialization process. With Handler, it allows to develop low level request & response. Both has it's own pros and cons.

There is no conclusion on which one is better. It goes back to you to choose which one is suitable for the project that you work on.

Posting data in JSON format to the ASP.NET website

When you are posting data using JQuery AJAX, you may call $.post() method to submit the data. We normally post the "data" by specifying the parameter name and value into the "data" parameter in the $.post() method. But, it will be very tedious to add new field to the data parameter when you already have lots of fields. To ease the coding maintenance, you may post the data in JSON format. This is quite simple by instantiating a new Object and set the property with values.

For example, I wrote this script in a HTML file:

    <script src="../Scripts/jquery-1.7.js" type="text/javascript"></script>
    <script type="text/javascript">
        function postJson() {
            var m = {
                myname: 'abc',
                myage: 10
                // add more properties here
            };
            var url = 'post_json.aspx';
            $.post(url, JSON.stringify(m), function (d) {
                if (d && d == 'ok') {
                    alert('ok');
                }
                else {
                    alert('failed');
                }
            });
        }
    </script>

Below is the "post_json.aspx" which you may use Handler (.ashx) to avoid the ASP.NET page life cycle.

using System.Web.Script.Serialization;

    public partial class post_json_post_json : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {      
            string json_text = this.Request.Form[0];
            JavaScriptSerializer js = new JavaScriptSerializer();
            CInfo my_obj = (CInfo)js.Deserialize(s, typeof(CInfo));

            this.Response.Write("ok");
            this.Response.End();
        }
    }

This is the data class that I'm using in this demo:

    public class CInfo
    {
        public string myname { get; set; }
        public int myage { get; set; }
    }

By using this technique, you will be achieving the following:

• Easy to maintain the JavaScript coding by replacing the data parameter in the $.post() with a stringify data in JSON format.
• Easy to maintain the code in ASP.NET - this is because you just need to declare a class which contains all the properties which is same as the data in JSON format. You may declare a property with other class type as well and also supporting object array.

Improving the web app by using AJAX + Handler

The following diagram explains the way to improve the web application response and reduces the server load by using AJAX call to the Handler.

For example, the "default.aspx" allows the user to post message and view all the messages. In the default.aspx, you will not use the GridView control to show all the messages. Instead, you make AJAX requests to the LoadMessage.ashx and show the result in the default.aspx. In this way, no viewstate and no ASP controls will be generated by LoadMessage.ashx.

The LoadMessage.ashx may return the result in 2 ways:

• formatted result (with HTML/TABLE tags or whatsoever) OR
• result in JSON format. Of course, the result in JSON format requies lesser bandwidth. But, you need to write some Javascript to format the result and show it in the default.aspx.

Another function provided by PostMessage.ashx will be a lot simpler. When the user type their message in the default.aspx (at the client site), you just need to retrieve the message from the screen and then make AJAX calls to the PostMessage.ashx (with the appropriate parameters).

Scaling out your system using web services

There are many different ways to improve the system respond time: one of the strategy is call scaling out. In our system design, we are implementing the "basic modules" (i.e., the system infrastructure) in web service and it is able to achieve this easily.

Web service does not have a huge different from a website. The noticeable different is that it does not have user interface (or web pages) for visitor to access. You may imagine that the web service like a mobile phone station/transmitter which is providing connection to the mobile phone. In .Net, the web service is implemented in ASMX format, WCF (Windows Communication Foundation) or simply ASPX (which returns the data in XML or JSON format).

Now, the best part of web service is not something that will make our solution cool. Instead, it is able to scale out to a server other than the web server (that is hosting the website). In this case, our customer might end up with a web server to host the website, a few web servers that host the web services.

Many programmers argue that web service is slow because of XML SOAP involves in the communication. It's true that web service in ASMX is slow but the visitor won't really feel it because of these communications were made among the web servers which is sitting next to each other. Of course, the web service implemented in XML SOAP is not suitable for real-time application. The real-time application requires low level socket programming and the handling strategy will be different.

Reusable class - user access control

This is another important area in the systems and it is reusable as long as the design is generic to cover all possible combination.

Basically, it should meet the following requirements:

• Security control - allows user to logon to the system and verify their user name and password.
• Access permission - permission could be controlled by menu level, screen level and field level (whether the field is show/hidden or enable/disable).
• User access log - this is compulsory for audit purpose and also the alert.
• Access deny alert - upon hitting certain number of invalid user name or password, the system should generate alert and email it to the system administrator.
• Password policy - you may consider to implement minimum password length and password complexity.
• Allow supervisor overriding - this can be useful when the current user does not have permission to access certain feature (such as edit customer address) but requires to do so.
• In case the system is a web base system, it should store the browser type and visitor IP address for audit purpose.

 You need the following classes to support the above mentioned requirements:

1. CUser - this class stores the user information such as user name, password, email address, etc.
2. CAction - this class stores all the permissions (or features) for the system.
3. CUserAccessLog - this class responsible for keeping track the user login and logout activities.
4. CLog - this class (which has been discussed in previous article) which is responsible for storing the audit information such as which "action" (or "featuer") the user has clicked.

For the detailed implementation, you might have work it out by yourself.

Reusable class - System log - the information for troubleshooting

In our system design, we are trapping all the exceptions that raised up at runtime and store it into a central database. This allows us to provide faster response time in fixing the error before the user complains. I know many people will ask why there is an error and why it was not catch at the development or testing phase. The answer is simple, we are not developing standard package. Almost all projects come with different requirements except for the "infrastructure" (such as the security, application log, etc).

To log down the exception, this error logging process should done in a very short time so that it won't affect the system performance or other processes. This can be achieve by saving the exception using a different thread.

In our system logging class, we have the following methods:

• AppendLog - this save the log as "audit log".
• AppendError - this save the exception and the failure point as "error".
• AppendWarning - this save the log as "warning" and it is very useful when the system setting is missing or misconfigure.

To track the failure point, refer to System.Diagnostics.StackTrace class.
To save the log information in another thread, refer to System.Threading.ThreadPool.

Our application log table design:

http://sqllauhw2000.blogspot.com/2012/07/you-need-application-log-for-your.html

Executing a process periodically in Windows

There are many types of application running in an enterprise. One of it is called the "nightly job" (i.e., the job that runs periodically) which is running on every night. On the other hand, some of the jobs is running hourly, weekly or monthly.

In our experience, the main issue in running a process periodically is that "schedule" itself. Consider this:

• The simplest schedule job - run the job every night.
• Run the job once for every hour, week, month or year.
• Run the job once for every hour, week, month or year between 10AM to 6PM.
• Run the job on every Monday, hourly between 10AM to 6PM.

This is complicated and requires lots of testing if you are creating your scheduler program. Instead, the better way is to utilized the Task Scheduler that comes with Windows. This is what you have paid for.

Below is the screen snapshot of where you can find the Task Scheduler.

Below are a few ways to execute a job/process periodically in Windows:

1. Develop a Windows Service - add a System.Threading.Timer class and execute the callback periodically. With Timer class, it is able to run the process repeated in the same interval. If you want to run the next callback in different timing, you will have to calculate the interval and call Change() which will be complicated (depends on your need). Another problem that might arise is how to include the dependencies (or DLL)? For example, you need to generate daily report in PDF format and email it to the bosses.
2. Develop a program (.EXE) and schedule it in the Windows Task Scheduler - this is simple and clear cut. With this option, it solves the scheduling problem. But, it does not solve the dependencies problem as mentioned above.
3. Develop a web page which execute the process in a different thread and schedule it in the Windows Task scheduler - another requirement for this option is that it requires a small program which will call the URL (using System.Net.HttpWebRequest class) and pass the URL as the command line parameter. This solves the scheduling complexity (which handles by the Windows Task Scheudler) and also the dependencies (where all the business processes/rules are stay in the ASP.NET website). Of course, if the business processes/rules were in the Windows Client EXE, you will have no choice but fall back to strategy #2 as mentioned above.

Open Data Protocol

Sometimes you might wondering how to develop module by module and then integrate all modules into a single system. Basically, you need a protocol that is design for the integration purpose. This might be something that you are looking for.

Check this out:

http://www.odata.org/developers/protocols/overview