May 18, 2015
Recap
In Part 1, we set up Twisted. In Part 2, we configured Twisted to run as a Windows service. Now, we will assign the right permissions to the account that the service is using.
Windows file permissions
Windows implements file-level security through access control lists. Windows provides the icacls utility to query and set permissions at the command line.
For example
icacls C:\PythonEnvs\Example /grant "nt service\my_service":(OI)(CI)RX
Let’s break this down:
C:\PythonEnvs\Example - The folder you want to change permissions on/grant - Grant rights additively"nt service\my_service" - The name of the virtual service account(OI)(CI) - Applies the grant recursively to this folder, subfolders, and filesRX - Grants Read and Execute access
Inheritance mode
Inherited folder permissions are specified as:
- OI - Object inherit - This folder and files (no inheritance to subfolders)
- CI - Container inherit - This folder and subfolders
- IO - Inherit only - The ACE does not apply to the current file/directory
These can also be combined:
- (OI)(CI) this folder, subfolders, and files
- (OI)(CI)(IO) subfolders and files only
- (CI)(IO) subfolders only
- (OI) (IO) files only
So BUILTIN\Administrators:(OI)(CI)F means that both files and subfolders will inherit ‘F’ (Full Control)
Similarly (CI)R means folders will inherit ‘R’ (Read folders only = List permission)
SS64 has a good reference for the icacls options.
Troubleshooting permissions
Many Windows admin tasks require the wonderful Sysinternals tools, written by Mark Russinonvich. The first thing I do on a new Windows server is download the Sysinternals Suite and put it on the system path. For this troubleshooting section, I’ll be using the psexec tool.
Whenever a service is not working, it’s helpful to try running it by hand, in the same user account and environment that the service is trying to run in. As far as I know, there is no way to run commands manually as a virtual service account (NT SERVICE\my_service). Put another way, there is no equivalent of su - <user> that can be used for virtual service accounts. However, we can approximate it. We can use psexec to run commands as the LocalService account (NT AUTHORITY\LocalService). It’s far from perfect, but if we can get it working under the LocalService account, chances are we can then get it working under a virtual service account.
Let’s look at some steps we can take to investigate permissions problems. First, open a new admin command prompt. On Windows 8 and newer, “Win-x then a” is a quick way to open an admin command prompt.
Here’s how to execute a single command, under the LocalService account.
psexec -u "NT AUTHORITY\LocalService" C:\PythonEnvs\Example\Scripts\pip freeze
The -u option tells psexec what user to run as. The rest of the line is the command that you want psexec to run. Notice that we pass the full path to the pip inside the virtualenv. pip freeze prints a list of the packages that are installed.
If you get a “Couldn’t install PSEXESVC service” error, try running psexec as administrator.
Here’s another example. We’re telling Python to import the pyodbc module, and print “ok” if it succeeds.
psexec -u "NT AUTHORITY\LocalService" C:\PythonEnvs\Example\Scripts\python -c "import odbc; print 'ok'"
Here’s how to start a new interactive shell in a new window, running as the LocalService user.
psexec -u "NT AUTHORITY\LocalService" -i cmd /k
It should look something like this.
In the screenshot above, I have used the whoami command to show that, in the new console window, we are running as NT AUTHORITY\LocalService.
If you want to start a new interactive session as a different user, but stay in your current command window, omit the -i argument to psexec.
psexec -u "NT AUTHORITY\LocalService" cmd /k
Since Windows does not print the username in the shell prompt, it can be hard to know what user you are running as. Use the whoami command to see the current user.
The whoami /all command shows a lot of useful information about the user, groups and privileges.
Two things to keep in mind when using psexec to troubleshoot. One, it must be called from an admin command prompt (Win-x then a). Two, you can not use psexec to run commands as a virtual service account.
When I’m setting up Windows service, I configure it to run under the LocalService account at first, and I use psexec to track down any permissions problems. Then I switch the service over to a virtual service account. I grant the virtual service account the same privileges that I granted to the LocalService account.
Zip files
You may find that the virtual service accounts you’re using to run the services don’t have permission to handle zip files. This may be due to a policy restriction set by your network administrator. The LocalService account may have the same restriction.
This is why, when I’m installing pyodbc, I have to pass the --always-unzip option to easy_install. Otherwise, easy_install installs it as a zip file, and the virtual service account doesn’t have permission to read it.
Another example: on Windows, distutils packages source distributions as zip files. Let’s say for example you’re running python’s simpleHTTPServer as a Windows service (using nssm), running it as account nt service\package_server. You can visit http://localhost:8000 and you get a nice file listing. However, if you try to download one of the zip files, you get a 404. Assigning permissions for nt service\package_server doesn’t work - the permissions apply to folders and text files but they don’t apply to zip files through inheritance. It only works if you assign read permission specifically for the zip file itself.
Here’s a command to specifically assign permissions for a zip file
icacls "C:\PackageServer\www\downloads\My_Package-0.0.1.zip" /grant "nt service\package_server":R
By your leave
We’ve come to the end of Part 3. The virtual service account under which the service is running has been assigned the permissions it needs.
Note that these commands can be used for any service you are running on Windows, not only those using Twisted or NSSM.
Thanks for following along, and find me on Twitter at @christianmlong if you have any suggestions or fixes.
Apr 21, 2015
Recap
In Part 1, we installed Twisted, and set it up to run inside a virtualenv. Now, we will configure Twisted to run as a Windows service, under a virtual service account.
Install NSSM
With some coding, Twisted applications can run as a Windows service. However, instead of doing that, I’m running my application under NSSM, the Non-Sucking Serivce Manager. This handy application allows you to run any command-line application as a Windows service.
Download NSSM and unzip it. In this example, I’m using NSSM version 2.24, so the folder name is nssm-2.24. Change these intructions as needed to match the version of NSSM you are installing. Move the nssm-2.24 folder to C:\Program Files. Add this to the system Path:
C:\Program Files\nssm-2.24\win64
Note that we are using the 64-bit version of NSSM, even though the application we are installing is running on 32-bit Python. The 64-bit version of NSSM can manage both 64-bit and 32-bit applications.
The nssm.exe file might be marked as untrusted. You can unblock it by right-clicking on the file and choosing Properties. In the Properties dialog, click Unblock.
More information about unblocking files in Windows.
NSSM command line
We will use the NSSM command line tools to configure the new service. The NSSM commands look like this:
nssm [nssm command] [service name] [arguments]
Type nssm help to get an overview of the commands. If you get a “not recognized” error from the shell, add NSSM to your path. Changes to the path don’t apply to already-open command windows, so open a new command window after changing the path.
Running nssm commands requires admin privileges.
Create the Windows service
The following commands assume that you have created a virtualenv called “Example”, as described in Part 1. Change these commands as needed to match the name of your service and your virtualenv. You have to quote paths if they contain spaces.
Start a new admin command prompt (Win-x then a):
nssm install my_service ^
C:\PythonEnvs\Example\Scripts\python.exe ^
"C:\PythonEnvs\Example\Scripts\twistd.py --python my_tacfile.tac"
For readability, I have broken this long command in to multiple lines using ^, the dos line-continuation character.
This will install a service called my_service.
Look at the paths in the command above. Instead of specifying the system-wide python.exe (C:\Python27\python.exe), we give the path to a python.exe in the Scripts folder of our virtual environment. This has the same effect as calling activate in an interactive session. The Python interpreter will have access to all the packages installed in that virtual environment.
We also have to specify the full path to twistd.py. This file comes with Twisted; it starts the server process.
However, we don’t need to specify the full path to my_tacfile.tac. This is because we specify a working directory for the Windows service, as covered in the next section.
The Windows service has been created, but we still need to configure it. Stay in the admin command prompt, and type:
nssm set my_service AppDirectory C:\PythonEnvs\Example\Lib\site-packages\my_app
In the above example, replace my_app with your app name (it’s the name you used when you did pip install). Check that directory; it should contain your .tac file.
By setting the AppDirectory config variable, we are telling NSSM to make that directory the current working directory before starting the service. That is why we did not need to specify the full path to my_tacfile.tac when we installed the service.
This is the equivalent of passing the --chroot option to twistd.
Set display name and description
nssm set my_service DisplayName "My App"
nssm set my_service Description "My sweet application - running as a Windows service."
The display name and description will show up in the Windows service Manager console.
nssm set my_service Start SERVICE_DELAYED_AUTO_START
This setting tells the Windows service to start automatically when the server restarts. The NSSM docs have more information about the possible startup options.
Which account should we use?
There are a number of accounts you can use to run your Windows service. It is a good idea to run network services under the least-privileged account possible. For that reason, a user account is not a good choice.
Windows provides some built-in accounts for this purpose:
-
The LocalSystem account (NT AUTHORITY\SYSTEM) is still quite privileged.
-
The NetworkService account (NT AUTHORITY\NetworkService) allows the service to access network resources on the Windows network. However, we don’t need to run as the NetworkService account if we are just serving local resources.
-
The LocalService account (NT AUTHORITY\LocalService) has traditionally been the account Windows sysadmins used for services. It does, however, have some drawbacks. There is only one LocalService account per machine. Let’s say you want to set up multiple services per server (a database and a web server, for example). If you assign permissions to the LocalService account for the benefit of one service, those permissions are shared by all the services that use that account.
None of these are ideal. We’re going to be using a different account type, covered in the next section.
Virtual service accounts
In Windows Server 2008 R2, Microsoft introduced a new kind of account for this purpose, called a “Virtual Service Account”. These accounts are automatically created, one for each Windows service. By default, they have few privileges. And, if you assign privileges to a virtual service account, those privileges apply only to that service and that account. Other services on the same machine do not get those privileges.
Virtual service accounts have the same name as the service they apply to. So, for a service called my_service, the account name would be NT SERVICE\my_service.
Virtual service accounts are not especially well documented. This Microsoft TechNet article is a good reference. There’s no conveniently-located anchor tag in that document, so scroll down to the section titled “Using virtual accounts”. Here’s another article. Again, scroll down to the section titled “Virtual Accounts”.
Ignore everything about “Managed service accounts”. Virtual service accounts and Managed service accounts are often lumped together in the documentation.
Set the account
Because of the advantages listed above, I use virtual service accounts to run my Twisted services on Windows 2012 R2. Virtual service accounts are available on Windows Server 2008 and later.
NSSM can not currently configure services to use virtual service accounts. I contacted the developer, and he said he is interested in adding support.
However, we can use good old sc to configure the service to use the virtual service account. sc is the service manager command line utility that comes with Windows.
sc config my_service obj= "NT SERVICE\my_service"
Yes, that is a space after the equals sign. sc is very particular. SS64 has a good reference for sc.
NSSM workaround
NSSM doesn’t know about virtual service accounts, so it will complain if you use it to configure a service that has been set to use a virtual service account. As a workaround, use sc to temporarily set the account back to LocalSystem.
For example:
Use sc to set the user to something NSSM understands
sc config my_service obj= "LocalSystem"
Use nssm to configure your service
nssm set my_service Description "A better description"
Use sc to set the user back to the virtual service account
sc config my_service obj= "NT SERVICE\my_service"
You don’t have to do it this way. Once you create the Windows service with NSSM, you can configure that service with the traditional Windows service configuration tools.
Start it up
This is what it should look like
C:\WINDOWS\system32>nssm start my_service
my_service: START: The operation completed successfully.
The service prorperties dialog should look like this:
Notice our service is running, and is set to start automatically.
Update
Updated 2015-05-19
Via email, Michael Schlenker points this out:
Virtual Service Accounts can have the ‘Logon as a Service’ right removed via
Group Policy, which leads to weird errors. It is actually a recommended step
for security hardening certain Windows servers (as you don’t want to have
random services running).
Good point, and thanks for the feedback.
At your service
We’ve come to the end of Part 2. We have a Twisted application running as Windows service. It is running under a virtual service account.
Of course, virtual service accounts can be used for any service you are running on Windows, not only those using Twisted or NSSM.
In the next part, we will configure the right privileges for the account. Thanks for following along, and find me on Twitter at @christianmlong if you have any suggestions or fixes.
Mar 25, 2015
Intro
Installing and configuring Twisted on Windows has gotten easier over the years. Here is how to do it in 2015, using the latest packaging tools.
Twisted runs well on Windows, in my experience. I moved some Twisted applications from Linux to Windows for a client, and they have been running happily for months now without issue.
These instructions have been tested with Windows 8.1 and Windows Server 2012 R2. The applications I run on Windows use Twisted.Web to serve single-page web apps, talking to a database using pyodbc. My ODBC driver is 32-bit, so I’m using 32-bit Python for these instructions. However, Twisted does not depend on pyodbc, so don’t install it if you don’t need it.
These instructions assume that you already have an application that runs on Twisted. The Twisted documentation has a good explanation of how to set up your project so it runs as a Twisted Applicaion. It also includes an nice selection of example applications, if you don’t have an existing Twisted application.
Install Python
Twisted requires Python 2. Install the latest version of Python 2.7 (direct link to the 32-bit installer). Run the installer, and in the installer dialog, choose the option to add python to the path.
Install a compiler
Some of Twisted’s dependencies have C extensions, and are not available from the Python Package Index in the binary wheel format. So, we need a compiler to compile them from source. This used to be tricky, but fortunately, Microsoft now provides a free download that makes it easy. Download the Microsoft Visual C++ Compiler for Python 2.7. It may have to be installed as admin, to get around policy restrictions on compilers.
Upgrade pip and virtualenv
The Python 2.7.9 installer now includes pip and virtualenv, and sets them up for you by default. However, it does not come with the very latest pip and virtualenv. Here’s how to upgrade them to the latest versions.
Start an admin command prompt. On Windows 8 and newer, Win-x then a is a quick keyboard shortcut to open an admin command prompt.
Upgrade pip to the latest version.
python -m pip install --upgrade pip
Upgrade virtualenv to the latest version.
pip install --upgrade virtualenv
Now close the admin command prompt. We will be installing the rest of the packages in to a virtualenv, and that does not require admin access. The great advantage of using a virtualenv is that it keeps the packages we install isolated from each other, and from the system-wide packages.
Set up a virtual environment
Start a regular (non-admin) command prompt. Win-x then c is a quick keyboard shortcut for a non-admin command prompt.
mkdir C:\PythonEnvs
virtualenv C:\PythonEnvs\Example
C:\PythonEnvs\Example\Scripts\activate.bat
This makes a new directory on the C: drive, makes a new virtualenv, and then activates the new virtualenv. You should see the name of the virtualenv in parentheses at the start of your command prompt, something like this:
When a virtualenv is activated, it looks for installed Python packages in its own site-packages directory C:\PythonEnvs\Example\Lib\site-packages, instead of looking in the system wide site-packages directory C:\Python27\Lib\site-packages. Note that we don’t need to be in the virtualenv directory for it to be active.
The virtualenvwrapper project is full of useful shortcuts for working with virtualenvs. For simplicity, I will be using only virtualenv, and not virtualenvwrapper, in this writeup. However, if you’re interested in setting up virtualenvwrapper, this patched version works on Windows with the latest version of virtualenv.
The state of Python packaging in 2015
Installing Python packages on Windows has gotten a lot easier over the years. The Python Package Index (PyPI) now provides pre-compiled binary installers in the wheel format for many packages.
When a wheel is not available, pip can automatically compile C extensions using this compiler that Microsoft provides at no cost.
However, there are still packages that are not available on PyPI. Many are distributed for Windows in the Windows installer format (.msi or .exe). Pip can not install these packages, but there is a way to install them in to a virtualenv.
Install Twisted
This will pull the latest version from PyPI. It will also install its dependencies. One dependency, zope.interface, will use the compiler to compile a C extension.
. . .
Installing collected packages: zope.interface, Twisted
Running setup.py install for zope.interface
building 'zope.interface._zope_interface_coptimizations' extension
. . .
If you get a vcvarsall error, install the Microsoft Visual C++ compiler.
Install dependencies from PyPI
Install pywin32.
As of March 2015, pywin32 is available on PyPI in the wheel format. That means it can be installed by pip. Note that in order to get the PyPI version, we must tell pip to install package pypiwin32, not pywin32.
Install dependencies that are not on PyPI
For packages that are not on PyPI, the installation steps are different. If the package is distributed using a Windows binary installer (.msi or .exe) we can use the older easy_install command to install it in to a virtualenv.
One such package is pyodbc, which my application uses to talk to the database. Twisted itself does not depend on pyodbc, so there is no need to install it if your application doesn’t use it.
As of March 2015, pyodbc is not available in wheel format from PyPI.
Download the Windows installer. Make sure to get the installer that matches your version of Python and your architecture. I am using this one “3.0.7 32-bit Windows Installer for Python 2.7”.
Use easy_install to install pyodbc in to the virtualenv from the executable Windows installer file.
easy_install --always-unzip C:\Path\to\pyodbc-3.0.7.win32-py2.7.exe
I’ll talk about why we need --always-unzip in Part 3.
Not all installers will work with easy_install this way. See this Stack Overflow question for more details.
Twisted Application
In this series, I’m assuming you have your project structured as a Twisted Application. In that structure, you have a .tac file that is the connection point between your code and the Twisted server, twistd.
Here a basic .tac file, taken from the documentation for Twisted Application.
service.tac
# You can run this .tac file directly with:
# twistd -ny service.tac
"""
This is an example .tac file which starts a webserver on port 8080 and
serves files from the current working directory.
The important part of this, the part that makes it a .tac file, is
the final root-level section, which sets up the object called 'application'
which twistd will look for
"""
import os
from twisted.application import service, internet
from twisted.web import static, server
def getWebService():
"""
Return a service suitable for creating an application object.
This service is a simple web server that serves files on port 8080 from
underneath the current working directory.
"""
# create a resource to serve static files
fileServer = server.Site(static.File(os.getcwd()))
return internet.TCPServer(8080, fileServer)
# this is the core part of any tac file, the creation of the root-level
# application object
application = service.Application("Demo application")
# attach the service to its parent application
service = getWebService()
service.setServiceParent(application)
If you don’t already have a project that you run under Twisted, the documentation has a nice set of examples to get you started.
Install your application
Whatever Twisted application you are going to be running on this server, install it as you normally would. For example:
Now that everything has been installed, check it.
should look something like this:
my_app==4.14.2
pypiwin32==219
Twisted==14.0.2
zope.interface==4.1.2
other dependencies here
. . .
Newer versions of some of these packages may since have been released.
If you are installing using pip, make sure your .tac file is included in your distribution file. For exapmple, put this line in your MANIFEST.in file:
Documentation on MANIFEST.in.
Run it
Try it out. Make sure your virtualenv is activated, and type:
python C:\PythonEnvs\Example\Scripts\twistd.py --help
The virtualenv picks up the right Python path, but on Windows we have to specify the full path to the twistd.py file. This command should give you a nice help message and no errors.
Now try running your app under Twisted.
python C:\PythonEnvs\Example\Scripts\twistd.py ^
--rundir "C:\PythonEnvs\Example\Lib\site-packages\my_app" ^
--python "my_tacfile.tac"
For readability, I have broken this long command in to multiple lines using ^, the dos line-continuation character.
This command should print out some lines showing the twistd server starting up. Again, on Windows, we have to specify the full path to the app install directory when starting the twistd server. Go try out your app, and press Ctrl-c to shut down the server when you’re done.
Up and running
That’s it for Part 1. We have installed Python, set up a virtualenv, and gotten your Twisted app up and running. In Part 2, we will set up a Windows service to run the app, using the virtual service account that was introduced in Windows Server 2008. In Part 3, we will look at setting the right privileges for the app. In Part 4, we will package the app and its dependencies for deployment to test and production servers. Thanks for reading, and if you have any questions or suggestions, let me know. I’m on Twitter at @christianmlong.