PowerShell – Trap cmdlet errors.

Posted on December 29, 2011 by admin

Sometimes working with data can be challenging.  As a SQL Developer, creating ETL solutions, is our responsibility all this data makes it to our users.  Sometime data manipulation might be limited in SSIS, we are force to look for other alternatives.  In my case, I use PowerShell, and I need a way to trap some import processing errors during the execution of the “Invoke-SQLcmd” cmdlet.  Then, I will be able to investigate why some on my data wasn’t making it into my SQL tables.

I try to use PowerShell the “Try-Catch” to trap the error but it was wrong implementation to trap the error in the SQL Server PowerShell “Invoke-SQLcmd” command.  So, How can I trap the command errors?  Well, you can use “CommonParameters“, which are an additional set of parameters automatically provided by Windows PowerShell.  In these additional set of parameters you will find the “-ErrorVariable” parameter which allow you to stores the error information in a given variable.  The error variable you provide in any cmdlet will be overridden each time the cmdlet execute, unless you add an “+” in front of it so it can append to it and collect error information.

For more
Information about all “CommondParameters“, execute at the PS prompt:


Help About_CommonParameters -full

Now, my implementation was to use the “-ErrorVariable” parameter to trap and save the error message with the SQLquery script that failed. Also, I make sure to the variable to a null value after I saved the results to disk. In my scenario, the data imported could break my SQL query and I want to make sure I can identify these records.

So, I’m using the ErrorVariable parameter in my “Invoke-SQLcmd” command.  Here’s a code snippet sample of it:

$reccount = 1; $sqlerr = $null;

:

Foreach($item in $Data)
{
:

  $outfile = "C:\ImportError\TSQLQUERY_$($reccount).sql";
  $SqlQuery =  "Insert into $tablename ($tblfields) values($fldsvalue);";
  Invoke-Sqlcmd -Database $dbName -ServerInstance $ServerName `
    -Query $SqlQuery -ErrorVariable sqlerr;

  if($sqlerr -ne $null)
  {
     "/*Error Found - on record# [ $reccount ]:> `r`n"+$Error[0] + "*/ `r`n" `
       |  Out-File $outfile -Encoding ASCII;
     $SqlQuery | Out-File $outfile -Encoding ASCII -Append;
     "`r`n /* "+"".padleft(45,"=")+" */`r`n" `
       | Out-File $outfile -Encoding ASCII -Append;
     $sqlerr = $null;
  }
   :
  $reccount++;
}

Basically, I want to make sure my “$sqlerr” Error Variable is initialized as $null in both the beginning and after the error have been saved to the output file. Then, we can view later the Error message and the SQL script that error out to later view the output file displaying it in your SSMS or your favorite editor.

Keep in mind, this will not stop the errors from been displayed on screen.  But, you could use another Common Parameter: ” -ErrorAction SilentlyContinue” to prevent displaying the errors.

Sample output  file image:

I hope you find this code snippet useful!

Posted in PowerShell, SQL Server | Comments Off

PowerShell – Sum items in a CSV file.

Posted on December 21, 2011 by admin

The fact that PowerShell gives you the ability to a build tasks in many different ways, it’s sometimes intimidating.  But, you’ll find the way to improve your coding as time goes.  I try, the best I can, to keep things simple.

So, here’s another approach on How-To “Sum” items in a CSV file:

My Data

## sbData.csv:
# Date,Item,ItemType,Amount,Company
# 12/12/11,Ben Aipa 8'10,Longboard,1500,BenAipa
# 12/12/11,Walden 9'6",Longboard,1100,Walden
# 12/13/11,Channel Island 6',Shortboard,800,ChannelIsland
# 12/13/11,Mark Richards 6'8",Shortboard,750,Surftech
# 12/14/11,Walden 7'6",Funboard,800,Walden
# - End of CSV file -

The request is to sum the amount of the items group by Date and ItemType.  And here’s the solution:

## - Working process to Sum amount in a CSV file by grouping columns:
$gdata = Import-csv sbData.csv | Group-Object -Property Date,ItemType;

## - BUild your new object with the sum items:
[Array] $newsbData += foreach($gitem in $gdata)
{
$gitem.group | Select -Unique Date,ItemType, `
@{Name = 'SumAmount';Expression = {(($gitem.group) | measure -Property Amount -sum).sum}}
};

## - Build back the CSV file with the Sum items
$newsbData | Export-Csv NewsbData.csv -NoTypeInformation;
ii NewsbData.csv;

So, you can substitute the group items in ” -property “ parameter with the ones you want to be grouped on, then use  the “Select -unique …” to include the grouped items you want to list.  The magic happens when including the  “Measure” command against the already grouped items so you can sum the ‘Amount’ field.  Now, this code could become a template to sum up items when needed.

The Result:

# - My CSV new Output data:
# "Date","ItemType","SumAmount"
# "12/13/11","Longboard","2600"
# "12/13/11","Shortboard","1550"
# "12/14/11","Funboard","800"
# - end of file -

Happy PowerShelling!!

Posted in PowerShell | Comments Off

QuickBlog – Finding a value in a PowerShell Hash table – Simple!!

Posted on December 16, 2011 by admin

For this quick blog, let’s create a month hash object table containing the number and name of the month.  Then, we are going to query this Hash Table object for a specific month.

1. Creating our Month Hash Table:

$MonthHashTable = @{ `
[int] "1" = "January"; `
[int] "2" = "February"; `
[int] "3" = "March"; `
[int] "4" = "April"; `
[int] "5" = "May"; `
[int] "6" = "June"; `
[int] "7" = "July"; `
[int] "8" = "August"; `
[int] "9" = "September"; `
"10" = "October"; `
"11" = "November"; `
"12" = "December"; `
};

Notice in this example, we’ve assigned ‘Strong-typed’ values to most of the keys (month number) as Integer, and leaving the last three to default to a string data “data type“.

Keep in mind, in a Hash Table you have Keys, and Values.  So, we are going to search for a ‘Key’ in order to get the ‘Value’ so we can extract the Month name.

Here’s the Hash Table Syntax for finding the value using the key:
$yourHashTable.Item( [#key#] )

2. Lets go and Looking for the fourth and eleventh month name:

## ----------------------------------------
## - Getting "April":
## - Value stored as Integer for "April":
$mth = 4;

## - Bad, passing a string value when expecting an Integer
## -> no key found, nothing to return:
$MonthHashTable.Item("4");

## - Good, getting month name using Integer value
## -> keys found, a values are returned:
$MonthHashTable.Item(4);
April

$MonthHashTable.Item($mth);
April

## ----------------------------------------
## - Getting "November":
## - Value stored as Integer for "November":
$mth = 11;

## - Bad, passing an integer value when expecting a string
## -> no key found, nothing to return:
$MonthHashTable.Item(11);

## - Bad, passing an integer value when expecting a string
## -> no key found, nothing to return:
$MonthHashTable.Item($mth);

## - Value stored as String for "November":
$mth = "11";

## - Good, getting month name using String value:
## -> key found, a value is returned:
$MonthHashTable.Item($mth);
November

As the sample show, we need to make sure the variable value must have the proper “data type” so it can find the key.

Happy PowerShelling!!

Posted in PowerShell | Comments Off

SQLPASS PowerShell Virtual Chapter – “Extending T-SQL with PowerShell” Posted Slides & Scripts..

Posted on December 5, 2011 by admin

For all whom attended the SQLPASS PowerShell Virtual Chapter – “Extending Your T-SQL Scripting with PowerShell” session on Wednesday November 16th, I finally got it posted here.

My sincere Apologies for the long delay in posting my session slides and demo scripts.  I thought I lost the material after having a disk drive corruption but I was able to recorver it.

Please, click on the link :< download now>: from SkyDrive. (rename file *.zip.txt to *.zip before extracting)

Don’t hesitate to contact me if you have any issues downloading it.

Posted in PowerShell, SQL Server | Comments Off

YES!! Windows Management Framework (WMF/PowerShell) 3.0 CTP2 is here.

Posted on December 3, 2011 by admin

YES!!  It’s another great accomplishment from the Microsoft PowerShell Team.  Just released on December 2nd 2011 this CTP (Community Technology Preview) come with fixes and more enhancements.  Please CHECK out PowerShell ISE, is really showing some progress.

PowerShell ISE V3 CTP2 Customizable Themes

Get it here: http://www.microsoft.com/download/en/details.aspx?id=27548

First thing!!!  READ:> IMPORTANT!!  YOU NEED TO UNINSTALL WMF 3.0 CTP1!  Or, suffer the consequences.

But, it you’re one of the few experiencing Windows 8 Preview and using Hyper-V 3.0 then you can create another VM for a clean CTP2 installation experience.  Or, just use the VM shapshot to to back and forth between version.  So, take a pick.

This version comes with some documentation, so make sure to ready it.  Also, don’t be afraid to submit bug reports at: https://connect.microsoft.com/PowerShell/. (you need to have a Live ID)

For example, I just put in a suggestion to include the ability to “Comment-In” and “Comment-Out” block of code: https://connect.microsoft.com/PowerShell/feedback/details/711231/ise-v3-need-to-be-able-to-comment-a-series-of-lines-in-a-block-of-code

*Hint*Hint*:  

  1. When running scripts, I notice there has been changes to how the “Set-ExecutionPolicy” behaves.  I’m used to set my execution policy to “RemoteSigned“, but for this version now I had to changed it to “Unrestricted”. (more test to be done)
  2. Also, don’t forget to execute the “Update-Help” as soon as you open PowerShell.  This is working for most part.

There will be more to come soon.

Posted in Hyper-V, PowerShell, Windows 8 | Comments Off

Windows 8 Hyper-V 3.0 – My Personal 8 Tips for the Newbie

Posted on October 31, 2011 by admin

Well, Here’s some tips to those who are first timers with Windows 8 Server (or Client) Preview edition.  I have to say, after been using the previous version of Hyper-V, I’m very happy for what i’ve seen so far in Windows 8 Server with Hyper-V 3.0.  As I finally completed updating/rebuilding all my VM’s for upcoming PowerShell and SQL Servers presentations, I have compiled 8 tips that might help the newbies get started using Microsoft Virtualization Technology.

From the Hyper-V Manager console, click on the VM Connect option to work with your VM's.

Tip #1 – Memory is important

Yes, I totally agree with this one.  After I upgraded my memory from 4GB to 8GB, all my VM’s are working OK.  I have experience giving 1.5GB to a Virtual SQL machine is ideal for development.

VM Memory Setting

Tip #2 – Enabling Hyper-V

First, There’s some differences between Hyper-V 3.0 server and Client.  But, let me be clear, that every machine might be different.  In my case, I have a 4 year old HP Pavilion dv7-* Entertainment laptop with Virutalization option enabled. I created a dual boot Windows 8 Server and Client version.

On the Windows 8 Server enabling “Hyper-V” role using the new “Server Manager” Dashboard was easy.  But, in my case, I had to manually enable “Hyper-V Core” on my Windows 8 client by turning it “ON” in Windows Features.  Here’s the command:

C:\Dism /online /enable-feature /featurename:Microsoft-Hyper-V

Tip #3 – Create your Hyper-V 3.0 Virtual Network Switch

This is very Important and one this you need to setup your Virtual Network connections in either Server or Client. Open “Hyper-V Manager” and look the “Virtual Switch Manager”.

Hyper-V Virtual Switch Settings

As you can see in my picture, I have a Loopback adapter Virtual Switch (External), and a Wireless Virtual Swtich(Externa) both setup as Legacy Adapters).   These two combination works great when VM’s talk to each other and, at the same time, have access to the internet for Windows Update.  Keep in mind, you can disable the wireless by going into the VM Settings option, going into the “Legacy Network Adapter”, and changing the “Virtual Switch” to “not Connected”.

VM Settings menu - Changing Virtual Switch to "Not Connected"

Tip #4 – Don’t need a SAN Storage

FYI. Hyper-V 3.0 will allow you to create your VM’s practically anywhere.  In my case, I have an External USB 1.o TB Storage drive.  This is good new because you don’t need a SAN storage unit, and can save Vm’s outside your Physical server drives.

Tip #5 – Set VM’s TimeZone correctly

This might be only in my case, but after creating my VM’s and trying to connect them to join my Virtual Domain Controller, I experience connectivity issues between my VM’s due to their Time Zones not been the same.  So, If you have a Virtual Domain Controller, make sure both Time Zone machines are the same.

Here’s an interesting link on “Time Synchronization in Hyper-V” (by the “Virtual PC Guy’s Blog”) you may find useful: http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx

Tip #6 – Learn to use the VM Snapshot Feature

Yes! This feature is a life saver.  It can help you troubleshoot VM issues by you taking the snapshot of a VM machine before doing any irreversible updates that could force you start over to rebuild the machine over.  It’s perfect when creating multiple scenarios for in test machines and adding descriptions to it.  Also, you can also revert to a previous snapshot, or apply the changes.

List of a server Snapshot showing different stages during a SQL Server Installation

Adding a Description to your Snapshot

 

Tip #7 – Bringing Legacy Microsoft VM to Hyper-V3

If you want to move all your existing VM’s from: Virtual PC, Windows Virtual PC, and Hyper-V, just remember to uninstall all “Virtual Components”, or “Integration Services” to prevent any misbehaviour in your VM.  Then, you will be able to install the Hyper-V 3.0 “Integration Services” so that Hyper-V can manager your VM’s services such as: Snapshot, Start, Shutdown, or Save.

*hint*: I have experience some issues with Windows Servers 2008, and Windows 2003 SP1 after mvoing them to Hyper-V 3.0.  The mouse won’t work even after installing “Integration Services”.

Tip #8 – Start using Use PowerShell

Yes! YES! This is your opportunity to start using PowerShell.  You can user the Hyper-V module and start managing your VM’s.  Also, don’t forget that PowerShell comes with over 1695 and about  63 Modules.  Keep in mind,  This numbers will varies depending on the enabled Windows features and/or installed Server Applications containing PowerShell modules.

Here’s some one-liners command you may want to try get the estimate numbers of PowerShell commands and modules:

1. Get total number of available  commands and list their names:

(get-command -Module *).count; (get-command -Module *) | Select name;

2. Get total numbers of available (installed) Modules, and list their names:

(get-Module -ListAvailable).count; (get-Module -ListAvailable) | Select name;

I hope this information is helpful.  That’s it for now!

Posted in Hyper-V, PowerShell, SQL Server, Windows 8 | Comments Off

SQL Server PowerShell SMO – Simple way to Change SQL User Passwords

Posted on October 28, 2011 by admin

Here’s a simple PowerShell SMO code that shows you How-To change a SQL user password.  Keep in mind, SMO needs to be installed and the assemblies loaded before using this code.

To load SMO, you can:

1. Install the SQLPS Module (is using SQL Server 2012 “Denali”), or Chad’s Miller SQLPS module for SQL Server 2008/R2.

Import-Module SQLPS -DisableNameChecking

2. Or, Load the SMO Assemblies using the PowerShell V2 “AddType” command: (but carefull is you have multiple SQL Server versionsin the same box)

Add-Type -AssemblyName “Microsoft.SqlServer.Smo”

3. Or, use the still reliable the old PowerShell V1 load assemblies line using:

[reflection.assembly]::LoadWithPartialName(“Microsoft.SqlServer.Smo”)

In the following example show how to connect to a SQL Server using both the default Windows Authentication (with high Administrator privileges already set), or SQL Server Authenbtication to later change a user password.

*Hint*: If you want to include special characters, you need to use the single qoutes or PowerShell will think that it’s a variable name.

As you can see,  with just a few line of PowerShell SMO code you can start orchestrating a solution that can be applied to your SQL Server environments.

Just try it!

Posted in PowerShell, SQL Server | Comments Off

Creating a Standalone Windows 8 Server Virtual Machine with SQL Server 2012 CTP3

Posted on October 24, 2011 by admin

Here’s how I build my version of a standalone workgroup Windows 8 Server Virtual Machine(VM) with SQL Server 2012 (“Denal;i”) CTP3. When you create this VM, make sure to give enough memory.

How-To create a VM in Hyper-V Manager console:

I’m not going deep on this topic but the Hyper-V Manager GUI it’s easy to use.  I’m assuming you already got an *.ISO image of both: Windows 8 Server Preview and SQL Server 2012 CTP3.  If not you’ll have to find it at Microsoft website and/or your MSDN/Technet subscribtion.

Before Creating your new Virtual machine:

Keep in mind, you’ll need to setup you Hyper-V environment.  Meaning, if you’re using a laptop and/or a desktop computer (not a server),  still you need to make sure it meets the Hyper-V requirements or it won’t work.

So, at least you will need to use the “Virtual Switch Manager…” to assist you setting up your virtual network card to use by any VM you create.

Notice, in my environment, I have created three virtual network adapters:

  1. Wired Internet
  2. Wireless Internet
  3. Loopback

The first two adapters serve my purpose to be able to connect to my physical machine Internet connection so I can do windows update.  The loopback adapter is for my internal network connection to my virtual Domain Controller.

*Hint*: In order to allow your VMs to access your external wireless adapter, you need to enable in “Server Manager” the “Wireless LAN Service” feature before you create the virtual wireless adapter.

Now, you are ready to create a New Virtual machine, and just follow the wizard:

The “New Virtual Machine Wizard” will help you configured everything you need.   Make sure you create this VM with enough memory.  In my case I assigned 1.5GB of memory.

Opening your Virtual Machine Connection:

There’s two ways to open your VM Connection in your “Hyper-V Manager” console:

  1. By double-clicking at the actual virtual name.
  2. Or, double-click at the actual virtual machine preview pane at the bottom left side of the “Hyper-V” console.

Now that your connection GUI is open, it’s a good time to start doing a VM Snapshot in case you need to go back and troubleshoot in case of problems. Here’s some pictures on How-To create a VM Snapshot:

If a Snapshot box asking to add a Name to your snapshot, go ahead and do it.  This box only comes when there has been changes done to your VM.

Now, Ready for SQL Installation.

Installing SQL Server 2012 (“Denali”) CTP3:

After building the virtual server, if you try to immediately install SQL Server 2012, it won’t work.  And, when you try to run the setup.exe, you won’t have access to the “SQL Server Installation Center” to view the “Hardware and Software Requirements” information.

Check SQL Server Requirements:

So first, you may want to check this link to read about what’s required to install this new version of SQL Server: http://msdn.microsoft.com/en-us/library/ms143506(v=SQL.110).aspx

Need Role(s) and Feature(s):

In order to make my SQL Server installation work, I had to open Windows 8 “Server Manager” and follow the wizard to do the following steps:

  1. Install the Application Server Role (you can add more roles as you need during this process).
  2. Then, you need to add the following features: (again, you can add additional features as needed)
    a. Enable the .NET Framework 3.5
    b. Enable the PowerShell ISE

After doing this steps, then I was able to get the SQL Server setup to work and allow me to start my installation.  Use the VM Connect GUI to allow you to attached the SQL Server  *.ISO image for your VM to start the SQL Server Installation.

I’m not going to show all the SQL Server installation screens but here’s to show that I’m able to proceed with the installation.

On the previuos picture, notice that I selected most of the features to install except the for the two Distributed Replay services. At the same time, I took a live snapshot of my VM before the actual installation process.

Installation Completed Successfully:

Yes! I got my SQL Server 2012 installed without a glitch!

Testing SQL PowerShell:

Now, I immediately testing the PowerShell SQLPS module. I was so excited that forgot to do something first.  Here’s my result of my first try:

Yes! I forgot to set my “Set-ExecutionPolicy” to “RemoteSigned“, then close and reopen my Windows Console.  I also I was able to SQLPS.exe from SSMS Database option just to test that there’s no errors.  So, everything works at least for now.

Ooops! Except for PowerShell ISE.  Yes! If you try to do an “Import-Module SQLPS -DisableNameChecking” then you get an error:

Don’t Worry!  PowerShell ISE is not the only editor.  You can still use Notepad to create/modify your script(s).  Or, just try downloading one of the free community editors from: SAPIEN, and PowerGUI just to name a few.

To test SQLPS I use the following command line:

Invoke-SQLcmd -database ReportServer -Query “Select top 3 * from dbo.DBUpgradeHistory

Final comments:

I know I may have skip some steps but the bulk on How-To create Window 8 Server VM in shown in this blog.  One important thing to keep in mind, these are still Community Technologies Preview (CTP) and it will change.

So, Don’t be afraid to try it!  This is why we have the ability to create Virtual Machines in our own developement machines.  Again, take advantage of Hyper-V.

The opportunity we have is to learn from them, assist giving feedback, and MOST IMPORTANT, it help us to stay ahead in upcoming technologies.

Enjoy, Learn, and Collaborate!

Bonus!!… Bonus!!

If you want a HACK to fix the PowerShell ISE that here it is: (Hack provided by one of our PowerShell MVP’s – Joel “Jaykul” Bennett
http://HuddledMasses.org
http://PowerShellGroup.org
)

  1. Go to you x:\Windows\System32\WindowsPowerShell folder (make sure to access this folder “As an Administrator” or it won’t work)
  2. Create a blank text filename: powerShell _ise.exe.config (Yes! this file extension is “.config”)
  3. Then, add the following XML lines, and save the file when done:
(Updated: 10/26/2011 – I missed the “runtime” section)

<configuration>
<startup useLegacyV2RuntimeActivationPolicy=”true”>
<supportedRuntime version=”v4.0″ />
</startup>
<runtime>
<loadFromRemoteSources enabled=”true”/>
</runtime>
</configuration>

Reopen PowerShell ISE and try to use the “Invoke-SQLcmd” command.

That’s it!

Posted in Hyper-V, PowerShell, SQL Server, Windows 8 | Comments Off

SQL Saturday #86 Tampa BI edition: SSIS – Analyzing your data integrating PowerShell session

Posted on October 21, 2011 by admin

I’m excited to bring this new topic to #SQLSat86 Tampa for the BI community.  So, here’s a brief rundown of what I will be covering on Saturday, November 5th morning session:

Topic: SSIS – Analyzing your data integrating PowerShell

*Note: This is not an all PowerShell Session but it plays a big role in this SSIS solution.

What will be covering:

1. Some basic to intermediate SSIS and PowerShell.
2. How to pass arguments between PowerShell and SSIS steps.
3. How to use PowerShell to assist in analyzing our tables and/or Data.
4. How to Integrate PowerShell in our SSIS Solution (when needed).
5. Tool available to allow these two technologies to integrate in our ETL solution.

This is going to be an ALL DEMO session.

Requirements:

1. Have some interest in PowerShell.
2. Some basic SSIS ETL experience.
3. Willing to be open to new technologies.
4. Most Important: No PowerShell experience is not required.

I will Demo:

1. Passing Arguments between PowerShell Applications in SSIS.
2. Work with both Script Task and Script Components.
3. Use of both VB and C# (CSharp) .NET Scripting code.
4. Some PowerShell.

I hope yopu will enjoy this presentation

Posted in PowerShell, SQL Server | Comments Off

PowerShell working with SQLPS or SMO…

Posted on September 22, 2011 by admin

Today, while help @meltondba with his SQLPS question on the enumerating jobs history, @LaerteSQLDBA provided a oneliner to provide this result.  There’s one concern, should we use SQLPS instead of SMO.   For this answer I’m going to point out MSDN article regrading the future of SQLPS in upcoming SQL Server releases: http://msdn.microsoft.com/en-us/library/cc280450(v=SQL.110).aspx

This article states that “… This feature will be removed in a future version of Microsoft SQL Server.  Avoid using this feature in new development work, and plan to modify applications that currently use this feature. Use the sqlps PowerShell module instead. For more information about the sqlps module, see Import the SQLPS Module. …”

So, during this exercise I found out that I can out with the same number of line for either SQLPS and SMO.  Both giving me the same results.

Here’s some picture ilustrating basic code snippet of both SQLPS and SMO to get some SQL Server Jobs information:

SQLPS sample of getting jobs information

SMO sample of getting jobs information

This pictures shows, it look simple enough.  So, if you’re building SQLPS scripts, you can easily start transitioning your code to SMO.

Posted in PowerShell, SQL Server | Comments Off