PowerShell Core – Updating your SQL Server Linux Docker Containers Images

In this post I’ll be covering how to install some needed components, how to commit the changes, and create a revised images for deployment.

In recent event and meetings, I’ve been talking about how to work SQL Server Linux Containers Docker images. As these images get your container up-and-running quickly they lacks some tools that may be useful to complete the SQL Server configuration.

What’s missing?

The SQL Server images contains a small footprint of Linux Ubuntu 16.04 Operating System (OS) and is meant for quick deployment. The OS side the container need to be kept updated regularly.

At the same time, when you starts exploring inside the container, there still missing components you may want to use:

  • vim – for editing text files.
  • ifconfig – to check your network interfaces.
  • ping – to check IP-Address can be reachable across the network.
  • curl – for transfering data.

So, after you pull the docker image, create the container using “docker run …“, and then get to the container Bash session by using “docker exec -it …“. Remember the bash session only get you to the “root” level as there’s no users set on these containers.

## - First time setup: (for "server:2019-CTP2.2-ubuntu" and )
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=$SqlPwd01A' -e "MSSQL_PID=Developer" -p 1433:1433 --name sql2k19_CTP2.3 -d mcr.microsoft.com/mssql/server:2019-CTP2.3-ubuntu;

## - Display all active containers;
docker ps -a

At this point make sure the active container status should be in “Up” status. Now can proceed to update the container.

Installing Missing Components

To have access to the container we use the “docker exec …” command.  This command will allow to get access to the container “root” prompt.

## - Configuring your container:
docker exec -it sql2k19_CTP2.3 bash

The first thing I would suggest to do, execute the following to commands:

## - Updating OS:
apt update

apt upgrade

Notice if you try to execute: vim, ping, ifconfig, and curl are not installed in the container images.

Let’s proceed to install these component by executing the following command:

## - Installing additional components:
apt-get -y install \
curl \
vim \
iputils-ping \
net-tools \
powershell-preview

Also, it’s a good idea to create a Downloads folder in case to install other application(s).

## - Create Downloads folder in root:
mkdir Downloads
chmod 755 Downloads

Notice that PowerShell Core Preview was included with the other missing components.  PowerShell has become a great tool to have in a Linux environment.

PowerShell Core SQLServer Module

Although, this is optional but it doesn’t prevent you to include PowerShell Core Preview 6.2.0-RC1 with the SqlServer module which included the “Invoke-Sqlcmd” use by many administrator.  This is a great module to have in a SQL Server container image.

So, from the “root” prompt in the container open PowerShell Core Preview, then proceed to install the SqlServer module preview version 21.1.18095.

## - Open PowerShell Core:
pwsh-preview

## - Install SqlServer module preview:
Install-Module SQLServer -AllowPreRelease

This completes the essential for using PowerShell to help managing a SQL Server instance(s).

How About Anaconda?

We could install the latest version of Anaconda with Python 3.7 in our SQL Server container image.

## - Change directory to Downloads folder:
cd Downloads

## - Download Anaconda with Python 3.7:
wget https://repo.anaconda.com/archive/Anaconda3-2018.12-Linux-x86_64.sh

## - Install Anaconda with Python 3.7:
bash Anaconda3-2018.12-Linux-x86_64.sh

This will give us the ability to test Python scripts within the container.

Testing installed Components

We need to verify that all previously installed components are working. Go back to the container “root” prompt, and to execute the commands:

ifconfig
ping 127.0.0.1
vim ~/.bashrc
pwsh
sqlcmd

Now, executing “sqlcmd” command line will not work unless you add the path to the executable to the “root” ~/.bashrc file:

## - Need to include the path to SQLCMD command:
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc

## - Refresh ~/.bashrc:
source ~/.bashrc

## - Run Sqlcmd command:
sqlcmd -L localhost -U sa -P 'sapwd'
> select @@version
> go
> exit

This is a good indication that our *SQL Server container is active. And, now we got all missing components installed.

Now, we need to make sure we don’t lose out changes.

Creating your own SQL Server Docker image

This is an important step so you won’t lose the changes already made to the container.  Below are the brief step to follow:

## - Commit the container changes: (repository name must be lowercase but Tags are OK with uppercase)
## -> docker commit "<Get-Container_ID>" "<Image-name>":"<TAG name>"

docker commit "<Get-Container_ID>" sql2k19_ctp2.3_sandbox:CTP2.3-Version01

## - List images included the committed ones:
docker images

## - Stop Image before the Save step:
docker stop sql2k19_CTP2.3
docker ps -a

## - Save docker updated image:
docker save -o ./Downloads/sql2k19ctp23_sandboxVer01.tar sql2k19_ctp2.3_sandbox

The “docker commit …” command, you’ll provide both the image-name (all lowercase) and a TAG name (uppercase allowed). You can be creative in having an naming conversion for you images repositories.

It’s very important to save images after doing the commit. I found out that having an active container would be useless without an image.  As far as I know, I haven’t found a way to rebuild an image from an existing container if the image was previously removed.

Summary

Hope this brief run down on working with SQL Server Docker container images will get you started with modifying existing images for quick deployment.

One thing to keep in mind!

  • The SQL Server Container memory need to be 4GB minimum.
  • In Windows, if your’re using non-Hyper-V virtualization tools such as Virtualbox, the virtual machine memory need to be change to 4GB.
  • Also, when you are creating images, the virtual machine disk size default is 20GB. This may need to be increase unless you keep cleaning/removing images to make room.

Just layout what you need, commit, save and deploy your docker solution in your environment.

Keep learning about this amazing technology!

 

PowerShell Core – How to install SQLServer Module (Preview) in WSL – Ubuntu 18.04

You all have been following Aaron Nelson blog post on Invoke-Sqlcmd availability Cross-Platform in the SqlServer module then you all are probably have proceeded to download the PowerShell SqlServer.

At the same time, on March 5th,  PowerShell Core 6.2.0-rc1 (Release Candidate) was made available for download.
Go and get it!

The thing is, in order to use the Invoke-SqlCmd cmdlet, you need to use PowerShell Preview version 6.2.0-rc1 (or greater).

Now, SqlServer Module can be easily installed in all platforms but I found out that it won’t install in Windows 10 WSL Ubuntu 18.04.

So, What the issue with Windows 10 WSL – Ubuntu 18.04?

Normally, when working with modules in PowerShell Core, I always use the following Cmdlets: Uninstall-Module to remove the module and then Install-Module with the “-AllowPrerelease” parameter. This will work flawlessly, but I found out that it won’t installed it in WSL – Ubuntu 18.04.

I don’t know why but it was installing the non-preview version SqlServer module 21.1.18080.   So, the following PowerShell Core command line will force the installation of SqlServer module version 21.1.18095-Preview.

Install-Module sqlserver -RequiredVersion 21.1.18095-preview -AllowPrerelease -Force

Now, we can start writing PowerShell Core SQL Server scripts in our Windows Subsystem for Linux – Ubuntu 18.04.

PowerShell – Docker Setup for Windows 10 WSL Ubuntu 18.04 with VMware Workstation

The purpose of this blog post is to show how to setup Docker Community Edition in a Windows 10 with VMware Workstation to be use in Windows Subsystem for Windows (WSL).

There are a few blog post that helped me figure out what’s needed to get this to work and I’ll be sharing these links at the end of this post.

My current environment

My current environment consist of the following components:

  • Windows 10 Build 17763
  • VMware Workstation Pro 12
  • *Oracle Virtualbox 5.2
  • WSL – Ubuntu 18.04
  • SQL Server 2017 Developer Edition
  • Windows PowerShell (v5.1.17763.316)
  • PowerShell Core GA v6.3.1 (both Windows and Linux)
  • PowerShell Core Preview v6.2.0-preview.4 (both Windows and Linux)

*Note: This is not the latest version  of Virtualbox but it’s still supported.

Remember, the purpose of this environment is to build a “developer sandbox” that can allow me to learn and work with Docker containers.

What’s needed!

Because I’m using VMware Workstation instead of Hyper-V, there are a few things need to be in place to make this work. Windows 10 need to have the following:

  • All Hyper-V services need to be disable by using “System Configuration” tool.

  •  Install VMWare Workstation Pro. (https://www.vmware.com/products/workstation-pro.html)
  •  Install Oracle Virtualbox version 5.2. (https://www.virtualbox.org/wiki/Download_Old_Builds_5_2)

  •  Install from the Microsoft Store, WSL – Ubuntu 18.04.

  • And, make sure to run “sudo apt update” and “sudo apt upgrade” because images are not updated with latest components.

Installing PowerShell Components

Next, the following Docker components packages from Chocolatey need to be install using Windows PowerShell with administrator privileges:

* Install docker

choco install -y docker

* Install docker-machine-vmwareworkstation

choco install -y docker-machine-vmwareworkstation

Getting WSL Ready for Docker

Now, open the “WSL – Ubuntu 18.04” Linux console and execute the following *commands:

sudo apt update

sudo apt upgrade

*Note: You’ll need to run these two commands manually to keep your Linux distribution up-to-date.

At this point, follow the Docker installation instructions for “Docker-CE for Ubuntu 18.04“. But, in a nutshell, here’s the shortcut:

sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

sudo apt-get update

sudo apt install docker-ce

sudo usermod -aG docker maxt

exit

At this point. make sure to reopen the WSL linux console.

Setup Docker-Machine in Windows

Back in Windows PowerShell, the next steps show the way to have Docker work in “WSL – Ubuntu 18.04“. Starting with Windows PowerShell console, execute the following commands:

docker-machine --native-ssh create -d vmwareworkstation default
docker-machine create docker-host

These commands should complete without any errors. At the same time, two virtual machines: “default” and “docker-host” will be created and running in *Virtualbox.

*Note: These two *NEED* to be running in order for docker to work with WSL. At the same time, both VMware Workstation and Virtualbox need to be installed or this will not work

To check that for the Docker-Machine environment(s) are working, use the following command:

docker-machine ls

Next, execute the following command to write down “docker-host” environment results to be copied into the Linux user ~/.bashrc file.

docker-machine env docker-host
PS C:\WINDOWS\system32> docker-machine.exe env default
$Env:DOCKER_TLS_VERIFY = "1"
$Env:DOCKER_HOST = "tcp://192.168.220.xxx:2376"
$Env:DOCKER_CERT_PATH = "C:\Users\max_t\.docker\machine\machines\default"
$Env:DOCKER_MACHINE_NAME = "default"
$Env:COMPOSE_CONVERT_WINDOWS_PATHS = "true"
# Run this command to configure your shell:
# & "C:\ProgramData\chocolatey\lib\docker-machine\bin\docker-machine.exe" env default | Invoke-Expression

Open a “WSL – Ubuntu 18.04 console to edit the user “~/.bashrc” file, to add the following Docker variables:

## Added manually for Docker machine docker-host:
export DOCKER_HOST=192.168.99.xxx:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=/mnt/c/users/max_t/.docker/machine/machines/docker-host
export DOCKER_MACHINE_NAME=docker-host
export COMPOSE_CONVERT_WINDOWS_PATHS=true

sudo vim ~/.bashrc

Reopen the “WSL – Ubuntu 18.04 console.

Testing Docker in WSL

Now, I can test Docker in my “WSL – Ubuntu 18.04 console session. Open PowerShell Core console, and execute the following command to run the Docker Hello-World demo:

docker run Hello-World

This command download (or pull) the Docker image, then run the Hello-World container. If everything work as expected, then it will display the following text.

To check both Docker image(s) and/or container(s) in WSL , use the following commands: (Picture

# - Check for all pulled images in system:
docker images

# - Check the status of active containers:
docker ps -a

As you can see there no issues executing Docker command lines in Linux PowerShell Core.

To see the full list of docker command line help available click on the following link.

After all this is done! Docker working in my WSL environment.

Limitations

YES! There are limitations. This is a workaround on the issue of using Docker without Hyper-V. And, this will allow you to:

  • Pull images
  • Update containers
  • Save images

In my environment, I found limitations working with Docker Network using WSL which can impact Windows Docker-Machine VM “docker-host” interface. This issue can force you to rebuild both VM interfaces: “default” and “docker-host“.

Make sure to learn how to commit, save, and reload Docker images.  Don’t lose your changes!

So, if you have either VMware Workstation and/or Oracle Virtualbox, consider investing the time creating a Linux virtual machine and then install Docker CE.

Summary

We have accomplished setting up Docker containers in *Windows 10 “WSL – Ubuntu 18.04” using both Windows PowerShell and PowerShell Core in Linux. So, using Oracle Virtualbox v5.2 with VMware Workstation is a required component to make this work.

*Note: These post is meant for people to make Docker work in WSL Linux.

Also, if you’re familiar with PowerShell, Docker commands can execute without any issues. Now, I can use my favorite editor SAPIEN’s PowerShell Studio to build my automation scripts with docker commands.

What’s Next?

Try downloading other Docker images, like SQL Server 2017 and SQL Server 2019. This is the quickest way for providing a built solution using containers.

Learn about Docker Compose, and Kubernetes as these can be use in the Cloud environment as well.

Go and Explores the possibilities of provisioning solutions to your organization!

Resource links

Getting the latest Tools for PowerShell SQL Server Automation

You all know how important is to have the tool that can make our life easy do our system administration, and become a hero in our organization. Here’s a startup helper guide to get you going with some PowerShell and SQL Server tools.

What is available for automation!

For script automation we could install either or both version of PowerShell Core: (As of February 19th, 2019)

Here are some important PowerShell Modules to use for SQL Server management scripting:

  • *SQLServer – This module currently can be use on SQL Server 2017 and greater.
  • *DBATools – This a community supported module that will work with SQL Server 2000 and greater.
  • DBAReports – Supports for Windows SQL Server.
  • DBCheck – Support for Windows SQL Server.

*Note: This module is coming popular in cross-platform systems (non-Windows)

All of the above module can be downloaded from the PowerShell Gallery from the PowerShell console using the Install-Module cmdlet.

Install-Module -Name SQLServer -Force -AllowClobber;

Now, when working with older versions of SQL Server (2008->2017), you will find the SQLPS module is loaded during the SQL Server installation.

Just remember, since SQL Server 2017, Microsoft has change the PowerShell SQLPS module to SQLServer module downloadable from the PowerShell Gallery. This module is not available in PowerShell Gallery, only available during the SQL Server installation.

When PowerShell SQL Server Module can’t provide a script?

It won’t hurt to install the SQL Server Management Objects (SMO) library in case you want to be creative and start building your own SQL PowerShell scripts. This library is already available cross-platform, meaning that it will work in Windows, Linux and MacOS environments.

In this case, you can install the SQL Server SMO library “Microsoft.SqlServer.SqlManagementObjects” from the PowerShell Console using the Install-Package cmdlet.

Install-Package -Name Microsoft.SqlServer.SqlManagementObjects -AllowPrereleaseVersions;

Wait! There is more

As you already know, to manage SQL Server in Windows environment, we use the SQL Server Management Studio. But, this
application won’t work cross-platform.

So, the cross-platform option available is Azure Data Studio (February edition):

Don’t forget to include for following extensions:

What about Python?

By now you should already know that Python has been around for many year as cross-platform interpreted object-oriented high-level language. And, its popularity keeps increasing.

I would recommend to take a look at the Anaconda Distribution, and specifically the one with the latest version of Python (v3.7).

Download Anaconda for data science platform:

This installation will include *All* Python packages available to build an application.

And, Python can interact with PowerShell too!

Ah finally Containers!

Yes! Containers has become popular and can’t be ignored. It can be use in both Windows, Linux and any cloud environments. Go ahead to learn how to work and manage Docker containers.

Docker site to Download the Docker CE.

Don’t forget to check Docker Hub to find the latest Docker Container images available for download. And, you will need to create an account before downloading images.  The image below shows how-to search for the SQL Server image.

In Summary

As technology will keep improving, make sure stay up-to-date. This give us the opportunity to improve our job position and be of value for the organization that hire us.

Don’t forget to look for the nearest technology event in your areas, as this is the opportunity to learn for free and gain invaluable knowledge.

Invoke-SQLCmd minor issue running some SQL Stored-Procedures…

First time I notice something strange with “Invoke-SQLCmd”, I was when executing the system Stored-Procedure “SP_Who2” and got the following error message:

PS C:\Users\Max> $sqlWho = Invoke-Sqlcmd “SP_Who2”
Invoke-Sqlcmd : The pipeline has been stopped.
At line:1 char:24
+ $sqlWho = Invoke-Sqlcmd <<<<  “SP_Who2”
    + CategoryInfo          : InvalidOperation: (:) [Invoke-Sqlcmd], PipelineStoppedException
    + FullyQualifiedErrorId : SqlExectionError,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand

Invoke-Sqlcmd : A column named ‘SPID’ already belongs to this DataTable.
At line:1 char:24
+ $sqlWho = Invoke-Sqlcmd <<<<  “SP_Who2”
    + CategoryInfo          : InvalidOperation: (:) [Invoke-Sqlcmd], DuplicateNameException
    + FullyQualifiedErrorId : SqlServerError,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand

PS C:\Users\Max>

I was trying to create a collection of object with the results from my “SP_Who2” and got the error stating “.. A column named ‘SPID’ already belongs to this DataTable..”.   So, I end up scripting out the SP_Who2, finding there are two columns with the same name ‘SPID’ and PowerShell didn’t like it.  I made the change to rename one of the columns to be SPID2 and save the T-SQL script to my PowerShell script file. 

And, the next time the ran my “Invoke-SQLCmd”, I had no problems and got my results so I could manipulate my .NET objects.

So, this is to make you aware that you will experience this minor issues when executing some of the SQL system stored-procedures.  And, for those who wonder… is this a BUG??  I really don’t think so!!  Because, the issue is in some of the system stored-procedures.  Should I submit this issue to the SQL Team to fix all stored-procedures generating columns with the same name?  This could be a major and unnecessary task.  Anyway, you were served!!

Happy PowerShelling!!