AWS Datapipeline and Python Script

AWS Datapipeline and Python Script

I had a task to do and it was supposed to be done quickly. Then I got to know that there is a language which can do certain task efficiently and easily as compared to other languages and its called "Python".

Python has lots of powerful library to perform tasks quickly as it is like a scripting language.
Although, I hadn't had any knowledge of using this wonderful language but I thought of giving it a try and guess what!!! ,  it is similar to other languages like Java,C# and also need less time to do certain tasks as lot of libraries support in python.

Let me tell you I had a task in which I need to do some of the manipulations in AWS resources and save result in other AWS resources and python has a powerful library called Boto which is very easy to work on . Have a look at the boto library here

I was able to complete my work quickly using python  and created a python script and now there is some requirement to schedule this script so that this script runs daily at a certain time and perform its task.

As we are using heavily the AWS resources for our work so it was not the difficult task to choose AWS Datapipeline to do this work for us using EMR clusters.

So, now I have all of the resources - my script was ready and i can also schedule that script by using aws datapiplines but a question pop up in my mind whether I can schedule a python script using datapipeline or not.
FYI, I was also new on datapipeline.

I decided to research on that and after lot of effort -searching on internet ;) and various hit and trial on datapipeline options .I was successfully able to schedule my python script using boto library on aws datapipeline.


So, Here are some of the points to schedule python script on aws datapipeline, so that it would be easy for you guys :-
Step1: Have your python script ready.
Step2: AWS account and console.
Step3: Choose Datapipeline and start creating a datapipeline.
Step4: Choose source as EmrActivity and provide the S3 path of  your script in "input"
           and provide output path to another S3 bucket location.
Step 5: In order to run python from EMR cluster ,you need to add  "preStepCommand" : ""   .
Step 6:Choose EMR cluster and choose the desired configuration of the hardware.
Step7 : Schedule your job and you can also add preconditions so that datapipeline checks for precondition fulfillment before each run.
Step8: Setup logs in your S3 logs directory so that you can check problem in your job and debug issue using those logs.

Step8: Set SNS topics and subscribe for job completion and job failure notifications.

Finally, have fun and let other hard work to be done for you by datapipelines.  

Mutex-Simply

Mutex - I just came across a very simple definition and example of mutex from Threading in C#, by Joe Albahari, so I just thought of sharing it.

Mutex

A Mutex is like a C# lock, but it can work across multiple processes. In other words, Mutex can be computer-wideas well as application-wide.

Acquiring and releasing an uncontended Mutex takes a few microseconds — about 50 times slower than a lock.

With a Mutex class, you call the WaitOne method to lock and ReleaseMutex to unlock. Closing or disposing aMutex automatically releases it. Just as with the lock statement, a Mutex can be released only from the same thread that obtained it.

A common use for a cross-process Mutex is to ensure that only one instance of a program can run at a time. Here’s how it’s done:

class OneAtATimePlease

{

  static void Main()

  {

    // Naming a Mutex makes it available computer-wide. Use a name that's

    // unique to your company and application (e.g., include your URL).

    using (var mutex = new Mutex (false, "oreilly.com OneAtATimeDemo"))

    {

      // Wait a few seconds if contended, in case another instance

      // of the program is still in the process of shutting down.

      if (!mutex.WaitOne (TimeSpan.FromSeconds (3), false))

      {

        Console.WriteLine ("Another app instance is running. Bye!");

        return;

      }

      RunProgram();

    }

  }

  static void RunProgram()

  {

    Console.WriteLine ("Running. Press Enter to exit");

    Console.ReadLine();

  }

}

If running under Terminal Services, a computer-wide Mutex is ordinarily visible only to applications in the same terminal server session. To make it visible to all terminal server sessions, prefix its name with Global\.

Note : For more in depth knowledge you can go to link

Unit Test not running in VS 2010 after installation of VS2012.

Many of developers might have experienced this scenario of unit test not getting executing in VS 2010 after installation of VS2012 next to VS2010.

The simple solution to this problem is to install VS2010 SP1.

For more insight on the issue have a look on this link

Hope this helps :)

Prime Numbers Up to a range K

In order to find all of the prime numbers up to a range  K, we can use Sieve of  Eratosthene.Please find below the algorithm:-

isPrime[0] = false
isPrime[1] = false
for i = 2 to K do
    isPrime[i] = true
for i = 2 to sqrt(K) do
    if isPrime[i] then
        for j = i * i to K with step i do
            isPrime[j] = false

where you can consider isPrime as array of bool up to K. 

I find out about this interesting algorithm during a programming contest and it can be useful for fast calculation of all prime numbers up to a particular range.

For further reading , please refer to below link:

http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes 

Transaction got aborted ,WHY?

Transaction Exception: The operation is not valid for the state of the transaction.

There may be different reasons for transaction exception in distributed transaction when we are using TransactionScope in our code :-

1. Transaction may get timed out and can cause this type of exception.So, when facing this type of exception in our code we should first check whether the transaction can be completed in specified time or not. If not then we should increase the transaction time .

Options to increase the transaction time out can be easily find out by going to msdn sites.

2. Majority of time we got this type of exception when we are using TransactionScope and inside that TransactionScope ,we are opening some sql connection and executing some sql query.

 using (TransactionScope Scope = new TransactionScope(TransactionScopeOption.RequiresNew))
    {
        using(SQLServer Sql = new SQLServer(this.m_connstring))
        {
            //code for sql query .
        }
    }

There may be couple of reason to get this type of error in such situations:-

• Whenever we are calling some sql procedure and in sql procedure we might be using transaction due to which when any exception occurs in procedure then transaction gets rolled back and when we try to do other stuff inside the transaction Scope then we will get this error as transaction is already rolled back and we are trying to do operation using that invalid transaction.

We might not be able to find out the cause of this error on code side when exception occurs in database side whenever we use sqlreader in our code in try ,catch block (and in catch block we are eating exception).

So, beware and try to lookout for SQL reader in code when you face any such exception and then check stored procedure whether its throwing any exception or not.

You might also look for transaction log,which system  generates for any distributed transaction.Since its not in scope of this post ,I am not talking about that in this post.

I hope this post can be helpful to you guys when you face this type of transaction exception.

Interesting .Net, Web concepts

Interesting .Net and Web concepts

Gac:-  http://www.codeproject.com/Articles/4352/Demystifying-the-NET-Global-Assembly-Cache

Sessions in Asp.Net:- http://www.codeproject.com/Articles/32545/Exploring-Session-in-ASP-Net 


Values Type/Reference Type:- http://blogs.msdn.com/b/ericlippert/archive/2010/09/30/the-truth-about-value-types.aspx

Stack memory  in .Net:- http://blogs.msdn.com/b/ericlippert/archive/2009/04/27/the-stack-is-an-implementation-detail.aspx

Http concepts:-   http://www.jmarshall.com/easy/http/

TPL introduction:- http://www.codeproject.com/Articles/152765/Task-Parallel-Library-1-of-n

Single responsibility principle:- http://www.developerfusion.com/article/137636/taking-the-single-responsibility-principle-seriously/

Event vs delegates:- http://blog.monstuff.com/archives/000040.html

Serialization in .Net:-  http://msdn.microsoft.com/en-us/library/ms973893.aspx




Note :-I will keep on adding other interesting topics in c# ,.net and of web concepts.

Sockets overview

Sockets:Use it or Not 
Many of us have listened about socket  in our programming language but we may not have given enough attention to it.
Some basic questions comes in mind when we listen about socket :-
What is socket? Why should I use socket ? What are  the benefits of using socket ? Should I use socket or not in my coding ? Is there other alternatives to socket programming ? What are the other ways ,using which I can do coding  without need of using sockets?  And One important question, I haven't used sockets till yet in my programming career,so why should i bother of using it or pay attention to understand the process of socket communication?


These are some of questions which came to my mind also, when i first started doing some programming using sockets.So,I started doing research about this topic and found it very interesting . It is basically the base of  network communication in our programming world.

The socket concept is not specific to any programming language, it is basically same concept in c#,java,c++ and other known programming languages for network communication.

Here in this post ,I will talk about sockets in c#.
So ,
Q -What is socket?
Ans-Sockets is a method for communication between a client program and a server program in a network. A socket is defined as "the endpoint in a connection." Sockets are created and used with a set of programming requests or "function calls" sometimes called the sockets application programming interface (API). The most common sockets API is the Berkeley  interface for sockets. Sockets can also be used for communication between processes within the same computer.


That's the answer I found from a site which I think will give you a good definition to understand sockets and its use. Also ,refer to link for sockets in .net for further understanding of socket APIs in .net.

For other above questions answers, I would say we have WCF  in .net for network communication.Internally WCF uses sockets for network communication. So, we can say WCF is just a wrapper on socket programming  which gives us many more  facilities than network communication e.g security , reliability and other options when doing communication between services and clients.

Sending  and Receiving data on network (Communication)
Before sending data using sockets on network we need to convert data into streams.Here comes the other concepts serialization and deserialization in our programming language to convert data into stream and converted stream back into data objects. FYI, In WCF we have DataContractSerialization and in web services we use normally XmlSerialization. I will not go further into serialization and deserialization. You can easily found topics on serialization and deserialization by searching on Google but I am sharing msdn link on serialization and deserialization in .net.

So,our purpose to know what is socket and why should we use it.. is almost complete.
For more information on sockets, how they communicate and what type of data can be passed from one socket to another ,please refer the link.

Next.. in this socket series , I will post  a simple example of web server using socket... (And also a simple comparison  of our web server code with IIS.. web server.)


Please share your views on this topic and let me know .. what more you think .. i should add in this socket series.

Microsoft Report Viewer 2010

Using to Report Viewer 2010 in Asp.net web application

To upgrade to Report viewer 2010 in a asp.net application, one has to do follow following steps:

·         Install report viewer 2010 redistributable in your machine from following location

http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=6442

System requirement needed to install this report viewer package is provided on the link location.

·         Some configurations need to be done on web page where we want to use report viewer control.

Register and using report viewer control on the web page.

Web page should contain following lines of code:

·         Changes need to be done in web.config file to use report viewer 2010 and for more information on reportviewer 2010 ,see link:http://gotreportviewer.com/

Sample of web.config is shown below:

Microsoft.Build.Utilities.v3.5 & Report Viewer 2008

Microsoft.Build.Utilities.v3.5

I had to convert one of my Asp.net web applications from .Net 3.5 version to .Net 4.0 version i.e my task needs me to use only .Net 4 dll in replacement of .Net 3.5 dll. Now, I had to convert all of the related projects or libraries used by my web application to work with .Net 4.I had done the same by changing each of projects .Net framework to 4 from 3.5. Certainly, it’s a very easy task; even I got no sweat to do so in our Air-conditioned officeJ. After changing, .net framework of all the related projects to 4 from 3.5 and done with some cosmetic changes to my web application to be compatible with .net framework 4, I had changed my web application web.config( whatever changes one need to do in web.config can be found in msdn ).

After all these changes with my web application, I tested my application and it was working fine on my development machine. This is not the end of story of this blog, real challenge starts from this onwards. Now I have to test my application on clean windows 2K8 server with IIS7. Then, I deployed my web application there on IIS7 in windows 2k8 server. When I entered username and password to web application then application crashed and showed me an error that it failed to load Microsoft.Build.Utilities.v3.5 dll. Now, it was my time to get surprised, how can it (web application) fail to load a .net 3.5 framework dll? And If I had forgotten to convert some project from .net 3.5 to .net 4 version then it should had been caught by me while I was testing the application on my development machine(windows 7) which hadn’t had .net 3.5 version installed.

After that I checked my web application again , I checked all projects , all of them were already targeted to .net framework version 4.Then I suspected my web.config but there was no sign that I was trying to use any .net framework 3.5 dll or especially Microsoft.Build.Utilities.v3.5 dll. Then I got to know that IIS 7 has a feature from where I can enable .net framework 3.5 and I checked that feature but it was also disabled. Now, it was hard to believe about this abnormal behavior by my web application. Finally, I searched info about this dll on internet and get to know that the dll has something to do with SSRS Report Viewer.

I checked about the report viewer in my application and my application was using repot viewer 2008 and finally I got to know about the real culprit report viewer 2008.

Summary:-Report Viewer 2008 does use Microsoft.Build.Utilities.v3.5 dll.

To remove reference of culprit Microsoft.Build.Utilities.v3.5 dll, I upgraded report viewer from 2008 to 2010 J

Simple Asp.net page

Simple ASP.net page (using Visual studio)!!!

If you are going to start developing a web application in microsoft technology then asp.net is the platform you are looking for...

Before start writing code for a simple asp.net page, a question clicks on my mind .....

Q- As a beginner, what i would expect when i have to start coding to make a simple web page using asp.net technology?
A- I would like to see my web page running along with knowing some of the basic funda's of doing coding using asp.net at that time .

Some of the concepts to keep in mind before starting:
1 - We can use any language C# or VB to do coding in Asp.net as basic .net framework is there to handle language compatibility.

2 - Unlike a traditional desktop program (which users start by running a stand-alone EXE file), ASP.NET applications are almost always divided into multiple web pages. This division means a user can enter an ASP.NET application at several different points or follow a link from the application to another part of the website or another web server.

3 - ASP.NET File Types:-

• Ends with .aspx -> These are ASP.NET web pages.

• Ends with .ascx -> These are ASP.NET user controls. User controls are similar to web pages,except that the user can’t access these files directly. Instead, they must be hosted inside an ASP.NET web page. User controls allow you to develop a small piece of user interface and reuse it in as many web forms as you want without repetitive code.

• Ends with .asmx -> These are ASP.NET web services—collections of methods that can be called over the Internet.

• web.config -> This is the XML-based configuration file for your ASP.NET application. It includes settings for customizing security, state management, memory management, and much more.

• Global.asax -> This is the global application file. You can use this file to define global variables (variables that can be accessed from any web page in the webapplication) and react to global events (such as when a web applicationfirst starts).

• Ends with .cs -> These are code-behind files that contain C# code. They allow you to separate the application logic from the user interface of a web page.

4 -ASP.NET server-side controls:-
ASP.NET actually provides two sets of server-side controls that you can incorporate into your web forms.

• HTML server controls -> These are server-based equivalents for standard HTML elements.These controls are ideal if you’re a seasoned web programmer who prefers to work with familiar HTML tags (at least at first). They are also useful when migrating ordinary HTMLpages or ASP pages to ASP.NET, because they require the fewest changes.

• Web controls -> These are similar to the HTML server controls, but they provide a richer object model with a variety of properties for style and formatting details. They also provide more events and more closely resemble the controls used for Windows development.Web controls also feature some user interface elements that have no direct HTML equivalent,such as the GridView, Calendar, and validation controls.

Now, lets start to create a simple new web form in Visual Studio. To do this,
select Website -> Add New Item. In the Add New Item dialog box, choose Web Form, type a name for the new page (such as SimplePage.aspx), make sure the Place Code in Separate File option is checked,and click Add to create the page.


In the new web form, some of the basic code would be automatically included in the .aspx file.The code contains many elements one such is page directive.
The page directive gives ASP.NET basic information about how to compile the page.
It indicates the language you’re using for your code and the way you connect your event handlers.If you’re using the code-behind approach, which is recommended, the page directive
also indicates where the code file is located and the name of your custom page class.

<%@ Page Language="C#" AutoEventWireup="true"
CodeFile="SimplePage.aspx.cs" Inherits="SimplePage" %>

In an ASP.NET web form, the doctype gets second place, and appears just underneath the page directive.
The doctype indicates the type of markup (for example, HTML or XHTML) that you’re
using to create your web page. Technically, the doctype is optional, but Visual Studio adds it
automatically. This is important, because depending on the type of markup you’re using there
may be certain tricks that aren’t allowed. For example, strict XHTML doesn’t let you use HTML
formatting features that are considered obsolete and have been replaced by CSS.
The doctype is also important because it influences how a browser interprets your web page.


"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

Every XHTML document starts out with this basic structure (right after the doctype):

"<"html xmlns="http://www.w3.org/1999/xhtml" ">"
"<" head runat="server" ">"
"<" title ">" Untitled Page "<" /title ">"
"<" /head ">"
"<" body ">"
"<" /body ">"
"<"/html ">"

When you create a new web form in Visual Studio, this is the structure you start with.
Here’s what you get:

• XHTML documents start with the tag and end with the tag. This element contains the complete content of the web page.
  

• Inside the element, the web page is divided into two portions. The first portion is the element, which stores some information about the web page. You’ll use this to store the title of your web page, which will appear in the title bar in your web browser. (You can also add other details here like search keywords, although these are mostly ignored by web browsers these days.) When you generate a web page in Visual Studio, the section has a runat="server" attribute. This gives you the ability to manipulate it in your code (a topic you’ll explore in the next chapter).
  

• The second portion is the element, which contains the actual page content that appears in the web browser window.

In an ASP.NET web page, there’s at least one more element. Inside the element is a

element. The element is required because it defines a portion of the page that
can send information back to the web server. This becomes important when you start adding
text boxes, lists, and other controls. As long as they’re in a form, information like the current
text in the text box and the current selection in the list will be sent to the web server using a
process known as a postback.

As far as we are familiar with the basic structure of .aspx page in web application in asp.net now, we will see where can we do the coding in web form.

Writing Code :

The Code-Behind Class:-
When you switch to code view, you’ll see the page class for your web page.

using System;
using System.Data;
using System.Configuration;
using System.Linq;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;

public partial class SimplePage: System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
}
}

This simple page is the class where we can add logic(Although good practice is to write business logic code in some class library which can be referenced into your asp.net web application ) to do some simple work in our application.

Now in order to show some thing on the webpage, I am writing some code in Page_Load event :

public partial class SimplePage: System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
Response.Write("Hello World");
}
}

This code will display " Hello World " on your first web page when we run the website using F5 command in visual studio.

Finally its over ,your first web page is running....:)

* Please try to see the Page Life Cycle overview from http://msdn.microsoft.com/en-us/library/ms178472.aspx to go for next stage.


I hope this would prove somewhat helpful for you guys to get some conceptual view before running you first web page in asp.net technology.

Difference between two dates as x month and y days.

If you have to find the difference between two dates as x month and y days .

Let the two dates be "StartDate" and "EndDate" and x be the month and y be the days after taking difference between StartDate and EndDate.
int x,y;
x= MonthDifference(StartDate,EndDate);

where MonthDifference(startDate,EndDate) is a function :

public int MonthDifference(DateTime startDate, DateTime endDate)
{
int noOfYears = endDate.Year - startDate.Year;
int noOfMonths = (endDate.Month - startDate.Month);
noOfMonths = noOfMonths + (noOfYears * 12);
return noOfMonths;
}


//To calculate No of Days and Month ..


if (System.DateTime.DaysInMonth(EndDate.Year, EndDate.Month)== (EndDate.Day - StartDate.Day + 1))

{
x = x+ 1;
y = 0;
}
else
{
if(EndDate.Day < StartDate.Day)
{
y = System.DateTime.DaysInMonth(user.EndDate.AddMonths(-1).Year,
user.EndDate.AddMonths(-1).Month) -
StartDate.Day + EndDate.Day + 1;
x = x - 1;
}
else
{
y= EndDate.Day -StartDate.Day + 1;
}
}

In this way you can get the difference between two dates as x month and y days.

Thanks
Varun

Default parameters in c# 4

I just read about a very cool feature of "default parameter" in a webblog of a microsoft employee.I just want that others should also be aware of this feature so,I am included this in my post also.

Presently in c# 3.5 we all are just using the function overloading in order to provide the default or optional parameter functionality.e.g

public class oldclass
  {
      public string A{ get; set; }
      public string B{ get; set; }
      public string C{ get; set; }
    
      public oldclass(string a)
      {
          A= a;
          B= "B";
          C= string.Empty;
      }
      public oldclass(string a, string b)
      {     
          A= a;
          B= b;
          C= string.Empty;
       
      }
      public oldclass(string a, string b, string c)
      {
          A= a;
          B= b;
          C= c;
      }
  }

But now in c# 4 what we do is:-

public class newclass
  {
      public string A{ get; set; }
      public string B{ get; set; }
      public string C{ get; set; }
            
      public newclass(string a, string b="B", string c=string.Empty)
      {
          A= a;
          B= b;
          C= c;
      }
  }

if we initialize constructor using-> new newclass("A");
then values for A="A",B="B"and C="".

So,Its quite cool to include this feature in c# 4 so that now,we don't have to use function overloading for getting functionality of optional parameters.