Getting started

Before installing the Surfly appliance make sure that you have the following information handy:

  • the domain name under which the appliance should run
  • a certificate with complete certificate chain which is valid for domainname.com and *.domainname.com
  • DNS is configured in such way that both domainname.com and *.domainname.com point to the same IP address

DNS configuration can be achieved by creating the following A records:

"" A 1.2.3.4
"*" A 1.2.3.4

For example, if you want to install Surfly on https://cobrowsing.yourwebsite.com, the following domain names and more should be covered by the certificate:

  • https://cobrowsing.yourwebsite.com
  • https://ws.cobrowsing.yourwebsite.com
  • https://session.cobrowsing.yourwebsite.com
  • https://google-com-p.cobrowsing.yourwebsite.com
  • https://bing-com-p.cobrowsing.yourwebsite.com

Make sure that the certificate is in *.pem format and has the following structure:

-----BEGIN RSA PRIVATE KEY-----
    (Your Private Key)
-----END RSA PRIVATE KEY-----

-----BEGIN CERTIFICATE-----
    (Your Primary SSL certificate)
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
    (Your Intermediate certificate)
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
    (Your Root certificate)
-----END CERTIFICATE-----

Wildcard certificate is only valid for one level of subdomains. For example, the wildcard certificate *.cobrowsing.domainname.com is valid for ws.cobrowsing.domainname.com, but not for session.ws.cobrowsing.domainname.com

Use sslcheck to verify that the certificate is valid. Usage example: sslcheck verify -c <path to the certificate>

Before the installation can be started, you need to be registered in our deployment system. In order to do that, please send us your domain name. In response you will get a client_id and a client_secret.

System requirements

Make sure your server satisfies the following requirements:

  • CentOS 9 Stream or RHEL 9
  • 4 core 2.5 GHz CPU
  • 8 GB RAM
  • 60 GB Disk space
  • 100Mbps network connection

Outbound access requirements

Prior to installation, ensure that your system has outbound web(http/https) access to the following domains:

  • Surfly domains:
    • registry.surfly.com
    • storage.surfly.com
    • build-agent.surfly.com
  • Python domains:
    • pypi.org
    • files.pythonhosted.org
  • Access to the following OS repositories
    • CentOS 9
      • baseos
      • appstream
    • Red Hat 9
      • rhel-9-baseos-rhui-rpms
      • rhel-9-appstream-rhui-rpms
  • Access to the Surfly COPR repository: https://copr.fedorainfracloud.org/coprs/surfly
    • copr.fedorainfracloud.org
    • download.copr.fedorainfracloud.org
  • Access to GitHub to syncronize install repository
    • github.com
    • api.github.com

Required packages

The following packages are necessary and will be automatically installed by the setup script if they are absent:

  • bash
  • openssh-server
  • cronie
  • sudo
  • tar
  • pigz
  • curl
  • dnf-plugins-core
  • git
  • invenv
  • podman (version 4.9.4 or newer)
  • slirp4ns
  • postgresql-server
  • redis

Installation

  1. Request access to the repository with installation script from Surfly

  2. Configure the network interface

  3. Install and configure the firewall

  4. Use the following script to install dependencies and configure the server. Run this script as root user

    #!/bin/bash
    echo "[Installing dependencies...]"
    dnf update -y
    dnf install sudo vim git -y
    
    echo "[Creating client user...]"
    adduser client
    usermod -a -G wheel client
    echo "client ALL = NOPASSWD : ALL" > /etc/sudoers.d/client
  5. Log in as client user

    su client
  6. Clone installation repository using authentication token provided by Surfly:

    cd ~
    git clone https://<token>@github.com/surfly/install.git
  7. Configure Surfly installation. Create config.yaml file in ~/install directory with the following content:

    client_id: your_client_id
    client_secret: your_client_secret
    version: prod
    ssl_certs_dir: /certs

    Check Configuration section for all available options

  8. Start installation script:

    ./setup

    This setup command by default will check if the version currently running on the server is the same being requested and skip the installation if it is the case. This can be overwritten by using the --force flag.

After installation process is finished, you should be able to open Surfly dashboard on https://app.domainname.com

Check Proxy environment section for instructions how to configure the server to use proxy during the installation

Do not run ./setup script as root user (for example, using sudo command). It may result in different authentication and permission errors

Proxy environment

Please check the following steps that should be helpful to install Surfly application in the proxy environment. The example assumes that 192.168.5.1:8080 is an address of the proxy server

  1. Configure environment by adding the following lines in /etc/environment file

    export HTTP_PROXY=http://192.168.5.1:8080
    export HTTPS_PROXY=http://192.168.5.1:8080
    export FTP_PROXY=http://192.168.5.1:8080
    export NO_PROXY="localhost,127.0.0.1"
    export http_proxy=http://192.168.5.1:8080
    export https_proxy=http://192.168.5.1:8080
    export ftp_proxy=http://192.168.5.1:8080
    export no_proxy="localhost,127.0.0.1"

    Add both upper- and lowercase values

  2. Update ~/install/config.yaml file. Check Configuration for all available options

    ...
    environment:
      dep_http_proxy: http://192.168.5.1:8080
      dep_https_proxy: http://192.168.5.1:8080
      dep_ftp_proxy: http://192.168.5.1:8080
      dep_no_proxy: localhost,127.0.0.1
    ...

Self signed certificates

  1. Create a private key

    openssl genrsa -out yourdomain.com.key 2048

    Never share generated private key

  2. Create Certificate Signing Request (CSR)

    openssl req -new -sha256 -key yourdomain.com.key -subj "/C=NL/ST=Nord Holland/L=Amsterdam/O=Surfly BV Client/OU=Dev/CN=yourdomain.com/emailAddress=support@yourdomain.com" -out yourdomain.com.csr
  3. Sign the certificate

    Generated CSR is valid only for yourdomain.com which is not enough for Surfly. When signing the certificate, please include the following additional subject identities:

    subjectAltName=DNS:yourdomain.com,DNS:*.yourdomain.com
    

  4. Combine private key, signed certificate and all intermediate and root certificates in one file as described here

    You can use cat command to combine multiple files in one:

    cat yourdomain.com.key yourdomain.com.crt > yourdomain.com.pem

  5. Copy your root CA certificate to /etc/pki/ca-trust/source/anchors/

  6. Update system certificates

    sudo update-ca-trust extract
  7. Modify ~/install/config.yaml and add a command to append your root CA certificate to the certificate file which is used by Surfly to verify SSL connections

  post_commands:
    - sudo tee -a /opt/surfly/ats_certs.pem < path_to_your_root_certificate

Free certificates

You can generate free wildcard certificates with Let’s Encrypt and use them with your Surfly installation. These certificates can be generated using lego, the Let’s Encrypt binary client written in GoLang.

  1. lego installation:

    cd /tmp
    wget https://github.com/go-acme/lego/releases/download/v4.6.0/lego_v4.6.0_linux_amd64.tar.gz
    tar -xvf lego_v4.6.0_linux_amd64.tar.gz
    mv lego /usr/local/bin/lego
  2. For the next step, you will need to check the lego documentation about DNS Providers to check exactly how to run the command depending on the provider used. For this example, I’ll use the Constellix documentation.

    Use the following command to verify that TXT record was deployed

    sudo dnf install bind-utils
    dig -t txt _acme-challenge.yourdomain.com

  3. Generate the certificate

    cd ~/
    export DOMAIN=yourdomain.com
    export CONSTELLIX_API_KEY=key
    export CONSTELLIX_SECRET_KEY=secret
    lego -a --domains=${DOMAIN} --domains=*.${DOMAIN} --email=admin@mail.com -k rsa2048 --dns constellix --pem run

    Generated certificates are located in ~/.lego/certificates folder. You can check their information with:

    lego list
  4. Configure Surfly installation to use the certificates. Add ssl_certs_dir configuration option to ~/install/config.yaml, for example

    ...
    ssl_certs_dir: ["/home/client/.lego/certificates/"]
  5. Run ~/install/setup script to apply changes

Let’s Encrypt certificates must be renewed every 90 days. To renew a certificate using lego it’s simple. Run the command from step 3., but with renew instead of run at the end. Then procceed with the steps 4. and 5.

Configuration

Surfly supports many configuration options which you can set in ~/install/config.yaml file.

config.yaml file should have at least 3 fields:

client_id: abc
client_secret: a1b4
version: prod

config.yaml file uses indentation to indicate the code blocks. Please pay extra attention to the section like environment, email and others

Do not copy configuration settings listed below directly in your config.yaml file. Many options contain non-default values and may make your installation unusable

Surfly configuration settings

# `client_id` is the unique identifier of a client in our build system
client_id: abc

# `client_secret` together with `client_id` are used to authenticate a server in our build system
client_secret: a1b4

# `version` is used to specify the version of the build. Recommended value is `prod`
version: prod

# List of paths to the certificates (deprecated since v3.304)
# certificates: ["/home/client/mycert1.pem", "/home/client/mycert1.pem"]

# Directory with certificates in *.pem format
ssl_certs_dir: /certs

# List of the steps to skip during the installation process
skip: [
  "prepare_machine",  # do not update dependencies
  "installation",  # download a build, but skip installation process
  "backup_database", # do not create a database backup copy
  "backup_services", # do not create a services backup copy
  "backup",  # do not create any backup
  ]

# Use verbose output for the ansible playbook
verbose: false

# Use local build file instead of requesting it from the Surfly build server
build_file: /home/user/abc_def_ghi.xfc.tar.gz

# All variables from the section will be set as environmental variables
# on the server when running ./setup command
environment:
  # Size of Varnish cache (default: 1.5G)
  varnish_size: 2G

  # Minimum number of threads in varnish per worker (default: 100, number of workers: 2)
  varnish_min_threads: 200

  # Size of short-lived Varnish cache (default: 1G)
  varnish_transient_size: 1000M

  # varnish storage backend. Usually "malloc" or "file,<filepath>". Default: "malloc"
  varnish_storage_backend: malloc

  # Size of Redis cache (default: 1G)
  redis_size: 1000M

  # Redis host
  redis_host: 127.0.0.1

  # Redis port
  redis_port: 6379

  # Size of in memory Apache Traffic Server cache (default: 512M)
  ats_size: 512M

  # Connection string to PostgreSQL database
  # Examples:
  # - Connect over Unix socket:
  #   surfly =
  #
  # - Remote database:
  #   surfly = host=1.2.3.4 port=5432 user=surfly_app password=secret
  # Documentation: http://www.pgbouncer.org/config.html
  db_connect_string: surfly =

  # Allow Surfly to create a session on private resources. Note: if you enable
  # this option, make sure your firewall configuration is strict.
  # Can also be a space separated list of private IP addresses and subnets to be
  # allowed
  allow_private_resources: true

  # Space separated list of the DNS servers (if it is not specified, it is
  # populated from the /etc/resolv.conf file)
  nginx_dns_resolvers: 8.8.8.8 8.8.4.4

  # max number of connections per nginx process (both frontend and backend)
  nginx_max_connections: 1024

  # The number of processes serving proxy requests. The recommended value is
  # less or equal to the number of CPU cores
  proxy_workers: 4

  # The number of processes serving dashboard requests. The recommended value is
  # less or equal to the number of CPU cores and less than 20
  dashboard_workers: 4

  # The number of processes serving Session API requests
  api_app_workers: 2

  # Rate limit Surfly REST API requests, req/min per IP address
  rest_api_rate_limit: 100

  # Proxy settings
  dep_http_proxy: http://10.0.0.11:8000
  dep_https_proxy: http://10.0.0.11:8000
  dep_ftp_proxy: http://10.0.0.11:8000
  dep_no_proxy: localhost,127.0.0.1

  # Default localization for new users
  # The time zone has to specified as a name of an entry in the tz database
  # see https://www.iana.org/time-zones and https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
  default_time_zone: UTC
  # The language has to be specified as a two letter iso 639-1 language code
  default_language: en

  # Include to set default login page to Single Sign On instead of password based login
  default_login_sso: true

  # Specify allowed HTTP methods for proxy application (default: all allowed (when absent))
  valid_http_methods: GET POST HEAD

  # Authentication token to call session API (default: randomly generated token (when absent))
  # Note: Spaces are not allowed
  cobro_auth_token: your_cobro_auth_oken

  # Force podman containers to run as root user instead of rootless (default: false (when absent))
  force_privileged: false

  # Include to block all connections to dashboard backends. Default: ommited
  disable_dashboard: true

  # Error template directory path, this can be used to rebrand error response pages.
  # For more info see the section on Surfly error in rebranding
  error_template_dir: /home/client/error_templ

  # Multi-server setup
  #
  # Unique cobro server id
  cobro_server_id: 81

  # Region of the server. The value is a valid session option for preferred region to start a session on
  cobro_server_region: eu-west

  # Description of the server. Used in `/v2/servers/` REST API endpoint
  cobro_server_description: Server 81 in OVH

  # IP addresses assigned to the server. This is a string with arbitrary format, it only affects
  # the server description in the `/v2/servers/` REST API endpoint. You need to update this manually
  # if the actual IP is changed.
  cobro_server_ips: 78.47.179.204, 2a01:4f8:b0:a033::2

  # Autorization token for internal dashboard API
  dashboard_auth_token: your_auth_token

  # Space separated list of dashboard URLs for a cobro server to register in
  dashboard_urls: https://app.yourdomain.com

  ##

  # Report attempts to violate the Content Security Policy to this URL
  csp_report_uri: https://example.com/csp/

  # Set timezone on the server
  server_timezone: UTC

  # Use the branding settings of the company with this ID to style the login
  # page and as a default for users in companies without custom branding
  # settings
  default_branding_company: 1

  # Specifies bind address for haproxy. More information is available here
  # https://www.haproxy.com/documentation/hapee/latest/configuration/binds/syntax/
  haproxy_listen_on: ::

  # Specifies dashboard session cookie timeout, in seconds
  session_cookie_age: 1209600

# Specifies the endpoints used for the session-recorder feature
start_recording_endpoint: https://recorder.example.com/start
end_recording_endpoint: https://recorder.example.com/end

# License key for Pdftron document editor, without it default pdf.js viewer is used
pdftron_key: PDFTRONKEY

# If specified, it enables server-side rendering mode in Pdftron document editor
pdftron_server: https://editor.example.com/

# Enable screenshot functionality on the server by providing credentials to
# S3 bucket to upload screenshots to
screenshots:
  screenshot_s3_bucket: screenshot_bucket
  screenshot_aws_access_key_id: ACCESSKEY
  screenshot_aws_secret_access_key: SECRETKEY

# Enable async session history CSV extraction by providing credentials to
# S3 bucket to upload CSVs to (link to uploaded file will be sent via e-mail)
session_history_csv:
  csv_upload_aws_bucket: csv_bucket
  csv_upload_aws_access_key_id: ACCESSKEY
  csv_upload_aws_secret_access_key: SECRETKEY

# Indicate the time period, in seconds, after which a session is classified as inactive and terminated as a 'zombie'
# if there has been no activity in the session during that time.
collect_zombie_period: 120

email:
  # By default Surfly prints the content of the all outgoing emails to the logs.
  # Use `surflyapp.email_backends.SMTPBackend` to send emails via your own
  # SMTP server or `surflyapp.email_backends.MandrillBackend` to use your
  # own Mandrill account
  email_backend: "surflyapp.email_backends.ConsoleBackend"

  # Default `from` email address
  default_from_email: support@surfly.com


  # SMTP configuration
  #
  # For more information, check official Django documentation
  # https://docs.djangoproject.com/en/1.11/topics/email/#smtp-backend
  email_host: localhost
  email_port: 1025
  email_host_user: client
  email_host_password: password
  email_use_tls: true
  email_use_ssl: false

  # MANDRILL configuration
  #
  # API key for your Mandrill account
  mandrill_api_key: "your_mandrill_key"

# OAuth2 configuration for admission into session
oauth:
  oauth_server_name: yourdomain.com
  github_client_id: CLIENTID
  github_client_secret: CLIENTSECRET

# Configuration for the Vonage SMS API
sms:
  vonage_com_api_key: APIKEY
  vonage_com_api_secret: APISECRET
  vonage_us_phone_number: PHONENUMBER

# Install and configure dns cache. dnsmasq is installed as the caching server
# (http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html) and
# NetworkManager is configured to use it
dns_cache: false

# Email address to which notifications should be send to
notify_email: admin@yourdomain.com

# The section is used by a redundant server to synchronize state with master node (main server)
master_node:
  host: 192.168.33.11  # IP address of the master node
  user: client  # the script establishes SSH connection with this user
  notify: ["success", "failure"]  # Send notifications to `notify_email` with synchronization's results

# Override redirect URLs for specific domains, specify favicon and tab title of the dashboard for specific domains
domain_redirects:
  surfly.com:
    contact: https://www.surfly.com/contact/
    dashboard_favicon: https://docs.surfly.com/installation/favicon.ico
    dashboard_title: "Surfly"
    docs: https://docs.surfly.com
    error404: https://some-other-404-page.example.com
    home: https://www.surfly.com
    register: https://some-other-register-page.example.com
    successful_session_start: http://help.surfly.com/en/articles/5342170-what-defines-the-successful-start-of-a-session
    videochat_docs: https://docs.surfly.com/surfly-options.html#header-video

# HTTP header to use for agent IP checks configured in Company settings,
# e.g. `X-Forwarded-For`. Set this if you have a hardware
# firewall in front of Surfly. Only the first comma-separated IP is considered
# when reading the header's value. If not set, the actual source IP is used.
client_ip_header: X-Forwarded-For
# Custom urls to verify the server status. By default https://app.your_domain.com/healthcheck/ is used
# Please make sure the domain name used here is being resolved to the server being configured
# If you want to simplify this on test environments you can also use http://localhost:8017/healthcheck/
# But this is strongly unadvised in production environments, as it won't check the certificate being used for the
# domain used on the server, and it might result in false positives during the deploy
healthcheck_urls:
  - https://eu.surfly.com/healthcheck/

# Run commands after Surfly installation was successfully finished. A command might be formatted using
# values from config.yaml file:
post_commands:
  - logger "Version {config[version]} is deployed"
  - curl -s http://localhost:8003/CobroVersion | echo $(</dev/stdin)

# Lock a specific version of Surfly
version_lock: v3.100

# Console settings

# Email backend for Console
# base.email_backends.SMTPBackend - send emails via your own SMTP server
# base.email_backends.MandrillBackend - use your own Mandrill account
# base.email_backends.ConsoleBackend - print the content of the all outgoing emails to the logs
console_email_backend: "base.email_backends.ConsoleBackend"
# The remaining email settings are identical to those outlined in the 'email' section above.

# S3 credentials for storing Console UI assets (user avatars, company logos, etc.)
console_ui_assets_s3_access_key_id: "s3_access_key_id"
console_ui_assets_s3_secret_access_key: "s3_secret_access_key"
console_ui_assets_s3_endpoint_url: "s3_endpoint_url"
console_ui_assets_s3_bucket_name: "s3_bucket_name"
console_ui_assets_s3_region_name: "s3_region_name"

Usage

After the first installation is finished, you will be asked to create a new reseller account which you can use to log in to Surfly Dashboard. Created account will also have permissions to view License information

You can create new companies and invite agents via Surfly Dashboard or via REST API

It is recommended to have one reseller account on the server and create new clients (companies) via Dashboard interface or via REST API. If you need to create more reseller accounts please check Managing accounts section

Surfly on-premise installation does not include some of the functionality that is found on https://app.surfly.com website. For example videochat is disabled because it uses a 3rdparty service (not included in price). Please contact support@surfly.com for details

Please check our Documentation for more details on how to use and integrate Surfly

Email configuration

Surfly does not require you to configure email functionality to create a session. However, user activation process includes an activation link from the email

Contact support@surfly.com if you want to brand your emails

Mandrill

Edit ~/install/config.yaml file

client_id: abc
client_secret: your_client_secret
vaersion: prod
email:
  email_backend: "surflyapp.email_backends.MandrillBackend"
  mandrill_api_key: "abc123"

Mandrill requires you to verify the domain name of the server by receiving an email. You can set up MX record in your DNS configuration to use your main email server for receiving emails which were sent to the domain name used by Surfly:

cobrowsing.yourdomain.com. MX 10 yourdomain.com.

Smtp server

Edit ~/install/config.yaml file

client_id: abc
client_secret: your_client_secret
vaersion: prod
email:
  email_backend: "surflyapp.email_backends.SMTPBackend"
  email_host: youremailserver.com
  email_port: 1025
  email_host_user: client
  email_host_password: password
  email_use_tls: true
  email_use_ssl: false

email_use_tls and email_use_ssl are mutually exclusive. See https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-EMAIL_USE_TLS

Reinstall Surfly server

Create backups of essential data

  1. Create a database backup

    pg_dump -U surfly_app surfly | pigz > ~/$(date +"%Y-%m-%d").gz
  2. Copy ~/install/config.yaml file

    Depending on your setup you might also want to copy the certificates

Upgrade process

  1. Install a new Surfly server by following instructions from here

    Please check system requirements

    Before running ./setup command, copy config.yaml file to ~/install/ folder

  2. Restore database using the following command:

    ./setup restore_database -f <database backup file>

Managing accounts

There are two types of accounts: a reseller account and a company account. It is recommended to create one reseller account (normally it happens during the installation) on the server and use it to create company accounts

Creating a reseller account

./setup create_user

Registration is disabled on the server

Set a session invitation type for a company

./setup manage_command set_invite_by {COMPANY_ID} {email|sms}

Example:

./setup manage_command set_invite_by 67145 sms

Rebranding

Surfly error rebranding

Error Description Filename
method not allowed This occurs when the request method is not a valid method as specified in valid_http_methods configuration option method_not_allowed.html
service unavailable This error occurs when the proxy server is not available to take any request service_unavailable.html
session not found This error occurs when the session doesn’t exist session_not_found.html

Steps to rebrand the errors:

  1. Create the error templates that you want to rebrand
  2. Put all the error templates that you made in step 1 in single directory. For example the directory is /home/client/error_templ
  3. Set the absolute path of the above-mentioned directory in error_template_dir under environment in configuration option.

If a custom error template is not present then the default error template will be used.

Server update

To update the server run ./setup script from install directory. By default the setup script requests a personalized build of the latest available version of Surfly application

Every time ./setup command is executed, a backup copy of the database is created and current build version is saved

We recommend to update Surfly weekly

Backup and restore

Backup

By default ./setup script creates a backup every time before you install a new version. It is also possible to create a backup with the following command:

./setup backup

backup command creates a dump of the database and archives all services.

You can find all created backups in ~/install/backupv2/ folder

Remove old backup folders to save disk space

Restore

Restoring previous Surfly version is a two-step process. First, you need to list all available backups and copy version you want to restore. Second, pass the backup version to the restore command

List available backups

To list all available backups, run ./setup list_backups command from the ~/install directory

Output sample:

Backup id            Surfly version                                     Db size, MB          Services, MB
2020-05-14.3         0efccd7d9ca0d52daec50ef3634c7767399f2451           88.9 KiB             70.9 MiB

list_backups command sorts output by the modification date. The most recent backup is shown last

Restore Surfly from the backup

To start restoring Surfly from the created backup, run restore command following by the version of the backup, for example:

./setup restore 2018-03-23.1

The command stops all running services, restores the database, installs required dependencies and restores Surfly services

Database

Surfly uses PostgreSQL as a database

By default Surfly creates a database backup every time you run ./setup command

You might need to stop/restart the following services before restoring the database:

systemctl --user stop ss-pgbouncer
sudo systemctl restart postgresql

After the database has been manually restored, start all services:

systemctl --user start ss-surfly.target

Backup database

pg_dump -U surfly_app surfly > ~/$(date +"%Y-%m-%d").sql

Restore database

dropdb -U postgres surfly && createdb -U postgres -O surfly_app -E UTF8 surfly && psql -U surfly_app surfly < ~/2018-03-16.sql

If the database file is too big, you can use pigz to create a compressed file:

Backup and compress database

pg_dump -U surfly_app surfly | pigz > ~/$(date +"%Y-%m-%d").gz

Restore database from a compressed file

dropdb -U postgres surfly && createdb -U postgres -O surfly_app -E UTF8 surfly && gunzip -c ~/2018-03-16.gz | psql -U surfly_app surfly

Upgrade PostgreSQL

~/install/upgrade_pg

Cache

Clear varnish cache

podman exec ss-varnish varnishadm -n /opt/varnish/ "ban req.url ~ ."

For older Surfly versions, you might need to run this command on the host instead:

varnishadm -n /opt/varnish/ "ban req.url ~ ."

Clear ATS cache

systemctl --user restart ss-trafficserver

Managing services

Surfly uses systemd for service management

Check status of all services

systemctl --user list-dependencies ss-surfly.target

Check status of a single service and view most recent logs

systemctl --user status ss-paws

Restart all services

systemctl --user restart ss-surfly.target

Stop all services

systemctl --user stop ss-surfly.target

Start all services

systemctl --user start ss-surfly.target

Failover

To provide failover capability for your Surfly setup, you will need at least 2 servers running the same version of Surfly application and a load balancer to handle failover.

To use a redundant server you need to register a new client_id

Load balancer

Surfly doesn’t require any specific load balancer to make failover work. Please check the configuration example for HAProxy:

global
  log 127.0.0.1 local0 debug
  user haproxy
  group haproxy

defaults
  log global
  mode tcp
  option tcplog
  maxconn 10000
  timeout connect 5s
  timeout queue   5s
  timeout client  40s
  timeout server  120s

# Make statistic available on /stat (Optional)
listen  stats
  bind 192.168.33.10:1936
  mode http
  log global

  maxconn 10

  stats enable
  stats refresh 5s
  stats show-node
  stats auth user:password
  stats uri  /stat

frontend lb_http
  bind :::80 v4v6
  mode http
  redirect scheme https if !{ ssl_fc }

frontend lb
  bind :::443 v4v6
  default_backend surfly_servers

backend surfly_servers
  fullconn 10000
  mode tcp
  server main 192.168.33.11:4433 check port 443 on-marked-up shutdown-backup-sessions send-proxy-v2
  server failover 192.168.33.12:4433 check port 443 backup send-proxy-v2

When main server 192.168.33.11 is down, all traffic is switched to the backup server 192.168.33.12.

Make sure that only the load balancer is able to establish connection on port 4433 with servers

Synchronizing Surfly version between main and redundant servers

We provide ~/install/sync_node.py script that synchronizes both the database and Surfly version from the main server to the redundant server

Set up automatic synchronization via cron job. Type crontab -e to create a cron job, for example:

SHELL=/bin/bash
PYTHONIOENCODING=utf8
PATH=/bin:/usr/bin:/usr/local/bin
0 3 * * * /home/client/install/sync_node_cron &>> /home/client/install/log/cron.log

To configure the synchronization process from main server to a redundant server, open ~/install/config.yaml file and add the following section:

master_node:
  host: 192.168.33.11
  user: client

Main server should be able to authenticate a redundant server via SSH key

You can configure ./sync_node.py script to verify the installation and send notifications to your email. Check notify_email option and master_node sections on Configuration page

Troubleshooting

You can contact Surfly by email support@surfly.com or call +31202611820

Changed Github RSA Key

For old installations that still used SSH keys to clone the installrepository, its possible that you will run into an issue related to Remove Host Identification when the repository tries to update itself. The error will look like this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Host key for github.com has changed and you have requested strict checking.
Host key verification failed.

This is related to a change Github made to update one of its private keys. You can find more information on their blog post: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

Their suggestion to fix this, you can simply remove the old key by running:

ssh-keygen -R github.com

After this, the next Surfly setup command should work normally.

Its also possible to make sure you have the current active keys after removing the old ones by running:

ssh-keygen -R github.com
curl -L https://api.github.com/meta | jq -r '.ssh_keys | .[]' | sed -e 's/^/github.com /' >> ~/.ssh/known_hosts

Surfly logs

Most of the Surfly logs end up in journalctl. You can follow them by running sudo journalctl -f.

Apache Traffic Server (ATS) logs are located in /opt/ts/var/log/trafficserver/

Ansible logs

Surfly uses Ansible to provision the server. In case of the provision script failure, it is possible to enable verbose logging by adjusting ~/install/config.yaml file:

...
verbose: true
...