Getting started

Before installing the Surfly appliance make sure that you have the following information handy:

  • the domain name under which the appliance should run
  • a certificate with complete certificate chain which is valid for and *
  • DNS is configured in such way that both and * point to the same IP address

DNS configuration can be achieved by creating the following A records:

"" A
"*" A

For example, if you want to install Surfly on, the following domain names and more should be covered by the certificate:


Make sure that the certificate is in *.pem format and has the following structure:

    (Your Private Key)

    (Your Primary SSL certificate)

    (Your Intermediate certificate)

    (Your Root certificate)

Wildcard certificate is only valid for one level of subdomains. For example, the wildcard certificate * is valid for, but not for

Use sslcheck to verify that the certificate is valid. Usage example: sslcheck verify -c <path to the certificate>

Before the installation can be started, you need to be registered in our deployment system. In order to do that, please send us your domain name. In response you will get a client_id and a client_secret.

System requirements

Make sure your server satisfies the following requirements:

  • CentOS 9 Stream or RHEL 9
  • 4 core 2.5 GHz CPU
  • 8 GB RAM
  • 60 GB Disk space
  • 100Mbps network connection

Outbound access requirements

Prior to installation, ensure that your system has outbound web(http/https) access to the following domains:

  • Surfly domains:
  • Python domains:
  • Access to the following OS repositories
    • CentOS 9
      • baseos
      • appstream
    • Red Hat 9
      • rhel-9-baseos-rhui-rpms
      • rhel-9-appstream-rhui-rpms
  • Access to the Surfly COPR repository:
  • Access to GitHub to syncronize install repository

Required packages

The following packages are necessary and will be automatically installed by the setup script if they are absent:

  • bash
  • openssh-server
  • cronie
  • sudo
  • tar
  • pigz
  • curl
  • dnf-plugins-core
  • git
  • invenv
  • podman
  • slirp4ns
  • postgresql-server
  • redis


  1. Request access to the repository with installation script from Surfly

  2. Configure the network interface

  3. Install and configure the firewall

  4. Use the following script to install dependencies and configure the server. Run this script as root user

    echo "[Installing dependencies...]"
    dnf update -y
    dnf install sudo vim git -y
    echo "[Creating client user...]"
    adduser client
    usermod -a -G wheel client
    echo "client ALL = NOPASSWD : ALL" > /etc/sudoers.d/client
  5. Log in as client user

    su client
  6. Clone installation repository using authentication token provided by Surfly:

    cd ~
    git clone https://<token>
  7. Configure Surfly installation. Create config.yaml file in ~/install directory with the following content:

    client_id: your_client_id
    client_secret: your_client_secret
    version: prod
    ssl_certs_dir: /certs

    Check Configuration section for all available options

  8. Start installation script:


    This setup command by default will check if the version currently running on the server is the same being requested and skip the installation if it is the case. This can be overwritten by using the --force flag.

After installation process is finished, you should be able to open Surfly dashboard on

Check Proxy environment section for instructions how to configure the server to use proxy during the installation

Do not run ./setup script as root user (for example, using sudo command). It may result in different authentication and permission errors

Proxy environment

Please check the following steps that should be helpful to install Surfly application in the proxy environment. The example assumes that is an address of the proxy server

  1. Configure environment by adding the following lines in /etc/environment file

    export HTTP_PROXY=
    export HTTPS_PROXY=
    export FTP_PROXY=
    export NO_PROXY="localhost,"
    export http_proxy=
    export https_proxy=
    export ftp_proxy=
    export no_proxy="localhost,"

    Add both upper- and lowercase values

  2. Update ~/install/config.yaml file. Check Configuration for all available options

      dep_no_proxy: localhost,

Self signed certificates

  1. Create a private key

    openssl genrsa -out 2048

    Never share generated private key

  2. Create Certificate Signing Request (CSR)

    openssl req -new -sha256 -key -subj "/C=NL/ST=Nord Holland/L=Amsterdam/O=Surfly BV Client/OU=Dev/" -out
  3. Sign the certificate

    Generated CSR is valid only for which is not enough for Surfly. When signing the certificate, please include the following additional subject identities:,DNS:*

  4. Combine private key, signed certificate and all intermediate and root certificates in one file as described here

    You can use cat command to combine multiple files in one:

    cat >

  5. Copy your root CA certificate to /etc/pki/ca-trust/source/anchors/

  6. Update system certificates

    sudo update-ca-trust extract
  7. Modify ~/install/config.yaml and add a command to append your root CA certificate to the certificate file which is used by Surfly to verify SSL connections

    - sudo tee -a /opt/surfly/ats_certs.pem < path_to_your_root_certificate

Free certificates

You can generate free wildcard certificates with Let’s Encrypt and use them with your Surfly installation. These certificates can be generated using lego, the Let’s Encrypt binary client written in GoLang.

  1. lego installation:

    cd /tmp
    tar -xvf lego_v4.6.0_linux_amd64.tar.gz
    mv lego /usr/local/bin/lego
  2. For the next step, you will need to check the lego documentation about DNS Providers to check exactly how to run the command depending on the provider used. For this example, I’ll use the Constellix documentation.

    Use the following command to verify that TXT record was deployed

    sudo dnf install bind-utils
    dig -t txt

  3. Generate the certificate

    cd ~/
    export CONSTELLIX_API_KEY=key
    export CONSTELLIX_SECRET_KEY=secret
    lego -a --domains=${DOMAIN} --domains=*.${DOMAIN} -k rsa2048 --dns constellix --pem run

    Generated certificates are located in ~/.lego/certificates folder. You can check their information with:

    lego list
  4. Configure Surfly installation to use the certificates. Add ssl_certs_dir configuration option to ~/install/config.yaml, for example

    ssl_certs_dir: ["/home/client/.lego/certificates/"]
  5. Run ~/install/setup script to apply changes

Let’s Encrypt certificates must be renewed every 90 days. To renew a certificate using lego it’s simple. Run the command from step 3., but with renew instead of run at the end. Then procceed with the steps 4. and 5.


Surfly supports many configuration options which you can set in ~/install/config.yaml file.

config.yaml file should have at least 3 fields:

client_id: abc
client_secret: a1b4
version: prod

config.yaml file uses indentation to indicate the code blocks. Please pay extra attention to the section like environment, email and others

Do not copy configuration settings listed below directly in your config.yaml file. Many options contain non-default values and may make your installation unusable

Surfly configuration settings

# `client_id` is the unique identifier of a client in our build system
client_id: abc

# `client_secret` together with `client_id` are used to authenticate a server in our build system
client_secret: a1b4

# `version` is used to specify the version of the build. Recommended value is `prod`
version: prod

# List of paths to the certificates (deprecated since v3.304)
# certificates: ["/home/client/mycert1.pem", "/home/client/mycert1.pem"]

# Directory with certificates in *.pem format
ssl_certs_dir: /certs

# List of the steps to skip during the installation process
skip: [
  "prepare_machine",  # do not update dependencies
  "installation",  # download a build, but skip installation process
  "backup_database", # do not create a database backup copy
  "backup_services", # do not create a services backup copy
  "backup",  # do not create any backup

# Use verbose output for the ansible playbook
verbose: false

# Use local build file instead of requesting it from the Surfly build server
build_file: /home/user/abc_def_ghi.xfc.tar.gz

# All variables from the section will be set as environmental variables
# on the server when running ./setup command
  # Size of Varnish cache (default: 1.5G)
  varnish_size: 2G

  # Minimum number of threads in varnish per worker (default: 100, number of workers: 2)
  varnish_min_threads: 200

  # Size of short-lived Varnish cache (default: 1G)
  varnish_transient_size: 1000M

  # varnish storage backend. Usually "malloc" or "file,<filepath>". Default: "malloc"
  varnish_storage_backend: malloc

  # Size of Redis cache (default: 1G)
  redis_size: 1000M

  # Redis host

  # Redis port
  redis_port: 6379

  # Size of in memory Apache Traffic Server cache (default: 512M)
  ats_size: 512M

  # Connection string to PostgreSQL database
  # Examples:
  # - Connect over Unix socket:
  #   surfly =
  # - Remote database:
  #   surfly = host= port=5432 user=surfly_app password=secret
  # Documentation:
  db_connect_string: surfly =

  # Allow Surfly to create a session on private resources. Note: if you enable
  # this option, make sure your firewall configuration is strict.
  # Can also be a space separated list of private IP addresses and subnets to be
  # allowed
  allow_private_resources: true

  # Space separated list of the DNS servers (if it is not specified, it is
  # populated from the /etc/resolv.conf file)

  # max number of connections per nginx process (both frontend and backend)
  nginx_max_connections: 1024

  # The number of processes serving proxy requests. The recommended value is
  # less or equal to the number of CPU cores
  proxy_workers: 4

  # The number of processes serving dashboard requests. The recommended value is
  # less or equal to the number of CPU cores and less than 20
  dashboard_workers: 4

  # The number of processes serving Session API requests
  api_app_workers: 2

  # Rate limit Surfly REST API requests, req/min per IP address
  rest_api_rate_limit: 100

  # Proxy settings
  dep_no_proxy: localhost,

  # Default localization for new users
  # The time zone has to specified as a name of an entry in the tz database
  # see and
  default_time_zone: UTC
  # The language has to be specified as a two letter iso 639-1 language code
  default_language: en

  # Include to set default login page to Single Sign On instead of password based login
  default_login_sso: true

  # Specify allowed HTTP methods for proxy application (default: all allowed (when absent))
  valid_http_methods: GET POST HEAD

  # Authentication token to call session API (default: randomly generated token (when absent))
  # Note: Spaces are not allowed
  cobro_auth_token: your_cobro_auth_oken

  # Force podman containers to run as root user instead of rootless (default: false (when absent))
  force_privileged: false

  # Include to block all connections to dashboard backends. Default: ommited
  disable_dashboard: true

  # Error template directory path, this can be used to rebrand error response pages.
  # For more info see the section on Surfly error in rebranding
  error_template_dir: /home/client/error_templ

  # Multi-server setup
  # Unique cobro server id
  cobro_server_id: 81

  # Region of the server. The value is a valid session option for preferred region to start a session on
  cobro_server_region: eu-west

  # Description of the server. Used in `/v2/servers/` REST API endpoint
  cobro_server_description: Server 81 in OVH

  # IP addresses assigned to the server. This is a string with arbitrary format, it only affects
  # the server description in the `/v2/servers/` REST API endpoint. You need to update this manually
  # if the actual IP is changed.
  cobro_server_ips:, 2a01:4f8:b0:a033::2

  # Autorization token for internal dashboard API
  dashboard_auth_token: your_auth_token

  # Space separated list of dashboard URLs for a cobro server to register in


  # Report attempts to violate the Content Security Policy to this URL

  # Set timezone on the server
  server_timezone: UTC

  # Use the branding settings of the company with this ID to style the login
  # page and as a default for users in companies without custom branding
  # settings
  default_branding_company: 1

  # Specifies bind address for haproxy. More information is available here
  haproxy_listen_on: ::

  # Specifies dashboard session cookie timeout, in seconds
  session_cookie_age: 1209600

# Specifies the endpoints used for the session-recorder feature

# License key for Pdftron document editor, without it default pdf.js viewer is used
pdftron_key: PDFTRONKEY

# If specified, it enables server-side rendering mode in Pdftron document editor

# Enable screenshot functionality on the server by providing credentials to
# S3 bucket to upload screenshots to
  screenshot_s3_bucket: screenshot_bucket
  screenshot_aws_access_key_id: ACCESSKEY
  screenshot_aws_secret_access_key: SECRETKEY

# Enable async session history CSV extraction by providing credentials to
# S3 bucket to upload CSVs to (link to uploaded file will be sent via e-mail)
  csv_upload_aws_bucket: csv_bucket
  csv_upload_aws_access_key_id: ACCESSKEY
  csv_upload_aws_secret_access_key: SECRETKEY

# Indicate the time period, in seconds, after which a session is classified as inactive and terminated as a 'zombie'
# if there has been no activity in the session during that time.
collect_zombie_period: 120

  # By default Surfly prints the content of the all outgoing emails to the logs.
  # Use `surflyapp.email_backends.SMTPBackend` to send emails via your own
  # SMTP server or `surflyapp.email_backends.MandrillBackend` to use your
  # own Mandrill account
  email_backend: "surflyapp.email_backends.ConsoleBackend"

  # Default `from` email address

  # SMTP configuration
  # For more information, check official Django documentation
  email_host: localhost
  email_port: 1025
  email_host_user: client
  email_host_password: password
  email_use_tls: true
  email_use_ssl: false

  # MANDRILL configuration
  # API key for your Mandrill account
  mandrill_api_key: "your_mandrill_key"

# OAuth2 configuration for admission into session
  github_client_id: CLIENTID
  github_client_secret: CLIENTSECRET

# Configuration for the Vonage SMS API
  vonage_com_api_key: APIKEY
  vonage_com_api_secret: APISECRET
  vonage_us_phone_number: PHONENUMBER

# Install and configure dns cache. dnsmasq is installed as the caching server
# ( and
# NetworkManager is configured to use it
dns_cache: false

# Email address to which notifications should be send to

# The section is used by a redundant server to synchronize state with master node (main server)
  host:  # IP address of the master node
  user: client  # the script establishes SSH connection with this user
  notify: ["success", "failure"]  # Send notifications to `notify_email` with synchronization's results

# Override redirect URLs for specific domains, specify favicon and tab title of the dashboard for specific domains
    dashboard_title: "Surfly"

# HTTP header to use for agent IP checks configured in Company settings,
# e.g. `X-Forwarded-For`. Set this if you have a hardware
# firewall in front of Surfly. Only the first comma-separated IP is considered
# when reading the header's value. If not set, the actual source IP is used.
client_ip_header: X-Forwarded-For
# Custom urls to verify the server status. By default is used
# Please make sure the domain name used here is being resolved to the server being configured
# If you want to simplify this on test environments you can also use http://localhost:8017/healthcheck/
# But this is strongly unadvised in production environments, as it won't check the certificate being used for the
# domain used on the server, and it might result in false positives during the deploy

# Run commands after Surfly installation was successfully finished. A command might be formatted using
# values from config.yaml file:
  - logger "Version {config[version]} is deployed"
  - curl -s http://localhost:8003/CobroVersion | echo $(</dev/stdin)

# Lock a specific version of Surfly
version_lock: v3.100

# Console settings

# Email backend for Console
# base.email_backends.SMTPBackend - send emails via your own SMTP server
# base.email_backends.MandrillBackend - use your own Mandrill account
# base.email_backends.ConsoleBackend - print the content of the all outgoing emails to the logs
console_email_backend: "base.email_backends.ConsoleBackend"
# The remaining email settings are identical to those outlined in the 'email' section above.

# S3 credentials for storing Console UI assets (user avatars, company logos, etc.)
console_ui_assets_s3_access_key_id: "s3_access_key_id"
console_ui_assets_s3_secret_access_key: "s3_secret_access_key"
console_ui_assets_s3_endpoint_url: "s3_endpoint_url"
console_ui_assets_s3_bucket_name: "s3_bucket_name"
console_ui_assets_s3_region_name: "s3_region_name"


After the first installation is finished, you will be asked to create a new reseller account which you can use to log in to Surfly Dashboard. Created account will also have permissions to view License information

You can create new companies and invite agents via Surfly Dashboard or via REST API

It is recommended to have one reseller account on the server and create new clients (companies) via Dashboard interface or via REST API. If you need to create more reseller accounts please check Managing accounts section

Surfly on-premise installation does not include some of the functionality that is found on website. For example videochat is disabled because it uses a 3rdparty service (not included in price). Please contact for details

Please check our Documentation for more details on how to use and integrate Surfly

Email configuration

Surfly does not require you to configure email functionality to create a session. However, user activation process includes an activation link from the email

Contact if you want to brand your emails


Edit ~/install/config.yaml file

client_id: abc
client_secret: your_client_secret
vaersion: prod
  email_backend: "surflyapp.email_backends.MandrillBackend"
  mandrill_api_key: "abc123"

Mandrill requires you to verify the domain name of the server by receiving an email. You can set up MX record in your DNS configuration to use your main email server for receiving emails which were sent to the domain name used by Surfly: MX 10

Smtp server

Edit ~/install/config.yaml file

client_id: abc
client_secret: your_client_secret
vaersion: prod
  email_backend: "surflyapp.email_backends.SMTPBackend"
  email_port: 1025
  email_host_user: client
  email_host_password: password
  email_use_tls: true
  email_use_ssl: false

email_use_tls and email_use_ssl are mutually exclusive. See

Reinstall Surfly server

Create backups of essential data

  1. Create a database backup

    pg_dump -U surfly_app surfly | pigz > ~/$(date +"%Y-%m-%d").gz
  2. Copy ~/install/config.yaml file

    Depending on your setup you might also want to copy the certificates

Upgrade process

  1. Install a new Surfly server by following instructions from here

    Please check system requirements

    Before running ./setup command, copy config.yaml file to ~/install/ folder

  2. Restore database using the following command:

    ./setup restore_database -f <database backup file>

Managing accounts

There are two types of accounts: a reseller account and a company account. It is recommended to create one reseller account (normally it happens during the installation) on the server and use it to create company accounts

Creating a reseller account

./setup create_user

Registration is disabled on the server

Set a session invitation type for a company

./setup manage_command set_invite_by {COMPANY_ID} {email|sms}


./setup manage_command set_invite_by 67145 sms


Surfly error rebranding

Error Description Filename
method not allowed This occurs when the request method is not a valid method as specified in valid_http_methods configuration option method_not_allowed.html
service unavailable This error occurs when the proxy server is not available to take any request service_unavailable.html
session not found This error occurs when the session doesn’t exist session_not_found.html

Steps to rebrand the errors:

  1. Create the error templates that you want to rebrand
  2. Put all the error templates that you made in step 1 in single directory. For example the directory is /home/client/error_templ
  3. Set the absolute path of the above-mentioned directory in error_template_dir under environment in configuration option.

If a custom error template is not present then the default error template will be used.

Server update

To update the server run ./setup script from install directory. By default the setup script requests a personalized build of the latest available version of Surfly application

Every time ./setup command is executed, a backup copy of the database is created and current build version is saved

We recommend to update Surfly weekly

Backup and restore


By default ./setup script creates a backup every time before you install a new version. It is also possible to create a backup with the following command:

./setup backup

backup command creates a dump of the database and archives all services.

You can find all created backups in ~/install/backupv2/ folder

Remove old backup folders to save disk space


Restoring previous Surfly version is a two-step process. First, you need to list all available backups and copy version you want to restore. Second, pass the backup version to the restore command

List available backups

To list all available backups, run ./setup list_backups command from the ~/install directory

Output sample:

Backup id            Surfly version                                     Db size, MB          Services, MB
2020-05-14.3         0efccd7d9ca0d52daec50ef3634c7767399f2451           88.9 KiB             70.9 MiB

list_backups command sorts output by the modification date. The most recent backup is shown last

Restore Surfly from the backup

To start restoring Surfly from the created backup, run restore command following by the version of the backup, for example:

./setup restore 2018-03-23.1

The command stops all running services, restores the database, installs required dependencies and restores Surfly services


Surfly uses PostgreSQL as a database

By default Surfly creates a database backup every time you run ./setup command

You might need to stop/restart the following services before restoring the database:

systemctl --user stop ss-pgbouncer
sudo systemctl restart postgresql

After the database has been manually restored, start all services:

systemctl --user start

Backup database

pg_dump -U surfly_app surfly > ~/$(date +"%Y-%m-%d").sql

Restore database

dropdb -U postgres surfly && createdb -U postgres -O surfly_app -E UTF8 surfly && psql -U surfly_app surfly < ~/2018-03-16.sql

If the database file is too big, you can use pigz to create a compressed file:

Backup and compress database

pg_dump -U surfly_app surfly | pigz > ~/$(date +"%Y-%m-%d").gz

Restore database from a compressed file

dropdb -U postgres surfly && createdb -U postgres -O surfly_app -E UTF8 surfly && gunzip -c ~/2018-03-16.gz | psql -U surfly_app surfly

Upgrade PostgreSQL



Clear varnish cache

podman exec ss-varnish varnishadm -n /opt/varnish/ "ban req.url ~ ."

For older Surfly versions, you might need to run this command on the host instead:

varnishadm -n /opt/varnish/ "ban req.url ~ ."

Clear ATS cache

systemctl --user restart ss-trafficserver

Managing services

Surfly uses systemd for service management

Check status of all services

systemctl --user list-dependencies

Check status of a single service and view most recent logs

systemctl --user status ss-paws

Restart all services

systemctl --user restart

Stop all services

systemctl --user stop

Start all services

systemctl --user start


To provide failover capability for your Surfly setup, you will need at least 2 servers running the same version of Surfly application and a load balancer to handle failover.

To use a redundant server you need to register a new client_id

Load balancer

Surfly doesn’t require any specific load balancer to make failover work. Please check the configuration example for HAProxy:

  log local0 debug
  user haproxy
  group haproxy

  log global
  mode tcp
  option tcplog
  maxconn 10000
  timeout connect 5s
  timeout queue   5s
  timeout client  40s
  timeout server  120s

# Make statistic available on /stat (Optional)
listen  stats
  mode http
  log global

  maxconn 10

  stats enable
  stats refresh 5s
  stats show-node
  stats auth user:password
  stats uri  /stat

frontend lb_http
  bind :::80 v4v6
  mode http
  redirect scheme https if !{ ssl_fc }

frontend lb
  bind :::443 v4v6
  default_backend surfly_servers

backend surfly_servers
  fullconn 10000
  mode tcp
  server main check port 443 on-marked-up shutdown-backup-sessions send-proxy-v2
  server failover check port 443 backup send-proxy-v2

When main server is down, all traffic is switched to the backup server

Make sure that only the load balancer is able to establish connection on port 4433 with servers

Synchronizing Surfly version between main and redundant servers

We provide ~/install/ script that synchronizes both the database and Surfly version from the main server to the redundant server

Set up automatic synchronization via cron job. Type crontab -e to create a cron job, for example:

0 3 * * * /home/client/install/sync_node_cron &>> /home/client/install/log/cron.log

To configure the synchronization process from main server to a redundant server, open ~/install/config.yaml file and add the following section:

  user: client

Main server should be able to authenticate a redundant server via SSH key

You can configure ./ script to verify the installation and send notifications to your email. Check notify_email option and master_node sections on Configuration page


You can contact Surfly by email or call +31202611820

Changed Github RSA Key

For old installations that still used SSH keys to clone the installrepository, its possible that you will run into an issue related to Remove Host Identification when the repository tries to update itself. The error will look like this:

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Host key for has changed and you have requested strict checking.
Host key verification failed.

This is related to a change Github made to update one of its private keys. You can find more information on their blog post:

Their suggestion to fix this, you can simply remove the old key by running:

ssh-keygen -R

After this, the next Surfly setup command should work normally.

Its also possible to make sure you have the current active keys after removing the old ones by running:

ssh-keygen -R
curl -L | jq -r '.ssh_keys | .[]' | sed -e 's/^/ /' >> ~/.ssh/known_hosts

Surfly logs

Most of the Surfly logs end up in journalctl. You can follow them by running sudo journalctl -f.

Apache Traffic Server (ATS) logs are located in /opt/ts/var/log/trafficserver/

Ansible logs

Surfly uses Ansible to provision the server. In case of the provision script failure, it is possible to enable verbose logging by adjusting ~/install/config.yaml file:

verbose: true