Getting started
Before installing the Surfly appliance make sure that you have the following information handy:
- the domain name under which the appliance should run
- a certificate with complete certificate chain which is valid for domainname.com and *.domainname.com
- DNS is configured in such way that both domainname.com and *.domainname.com point to the same IP address
DNS configuration can be achieved by creating the following A
records:
"" A 1.2.3.4
"*" A 1.2.3.4
For example, if you want to install Surfly on https://cobrowsing.yourwebsite.com, the following domain names and more should be covered by the certificate:
- https://cobrowsing.yourwebsite.com
- https://ws.cobrowsing.yourwebsite.com
- https://session.cobrowsing.yourwebsite.com
- https://google-com-p.cobrowsing.yourwebsite.com
- https://bing-com-p.cobrowsing.yourwebsite.com
- …
Make sure that the certificate is in *.pem format and has the following structure:
-----BEGIN RSA PRIVATE KEY-----
(Your Private Key)
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
(Your Primary SSL certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Intermediate certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Root certificate)
-----END CERTIFICATE-----
Wildcard certificate is only valid for one level of subdomains. For example, the wildcard certificate *.cobrowsing.domainname.com is valid for ws.cobrowsing.domainname.com, but not for session.ws.cobrowsing.domainname.com
Use sslcheck to verify that the certificate is valid. Usage example: sslcheck verify -c <path to the certificate>
Before the installation can be started, you need to be registered in our deployment system. In order to do that, please send us your domain name. In response you will get a client_id and a client_secret.
System requirements
Make sure your server satisfies the following requirements:
- CentOS 9 Stream or RHEL 9
- 4 core 2.5 GHz CPU
- 8 GB RAM
- 60 GB Disk space
- 100Mbps network connection
Outbound access requirements
Prior to installation, ensure that your system has outbound web(http/https) access to the following domains:
- Surfly domains:
- registry.surfly.com
- storage.surfly.com
- build-agent.surfly.com
- Python domains:
- pypi.org
- files.pythonhosted.org
- Access to the following OS repositories
- CentOS 9
- baseos
- appstream
- Red Hat 9
- rhel-9-baseos-rhui-rpms
- rhel-9-appstream-rhui-rpms
- CentOS 9
- Access to the Surfly COPR repository: https://copr.fedorainfracloud.org/coprs/surfly
- copr.fedorainfracloud.org
- download.copr.fedorainfracloud.org
- Access to GitHub to syncronize
install
repository- github.com
- api.github.com
Required packages
The following packages are necessary and will be automatically installed by the setup
script if they are absent:
- bash
- openssh-server
- cronie
- sudo
- tar
- pigz
- curl
- dnf-plugins-core
- git
- invenv
- podman (version 4.9.4 or newer)
- slirp4ns
- postgresql-server
- redis
Installation
Request access to the repository with installation script from Surfly
Use the following script to install dependencies and configure the server. Run this script as root user
#!/bin/bash echo "[Installing dependencies...]" dnf update -y dnf install sudo vim git -y echo "[Creating client user...]" adduser client usermod -a -G wheel client echo "client ALL = NOPASSWD : ALL" > /etc/sudoers.d/client
Log in as client user
su client
Clone installation repository using authentication token provided by Surfly:
cd ~ git clone https://<token>@github.com/surfly/install.git
Configure Surfly installation. Create
config.yaml
file in~/install
directory with the following content:client_id: your_client_id client_secret: your_client_secret version: prod ssl_certs_dir: /certs
Check Configuration section for all available options
Start installation script:
./setup
This setup command by default will check if the version currently running on the server is the same being requested and skip the installation if it is the case. This can be overwritten by using the
--force
flag.
After installation process is finished, you should be able to open Surfly dashboard on https://app.domainname.com
Check Proxy environment section for instructions how to configure the server to use proxy during the installation
Do not run ./setup
script as root user (for example, using sudo command). It may result in different authentication and permission errors
Proxy environment
Please check the following steps that should be helpful to install Surfly application in the proxy environment. The example assumes that 192.168.5.1:8080 is an address of the proxy server
Configure environment by adding the following lines in
/etc/environment
fileexport HTTP_PROXY=http://192.168.5.1:8080 export HTTPS_PROXY=http://192.168.5.1:8080 export FTP_PROXY=http://192.168.5.1:8080 export NO_PROXY="localhost,127.0.0.1" export http_proxy=http://192.168.5.1:8080 export https_proxy=http://192.168.5.1:8080 export ftp_proxy=http://192.168.5.1:8080 export no_proxy="localhost,127.0.0.1"
Add both upper- and lowercase values
Update
~/install/config.yaml
file. Check Configuration for all available options... environment: dep_http_proxy: http://192.168.5.1:8080 dep_https_proxy: http://192.168.5.1:8080 dep_ftp_proxy: http://192.168.5.1:8080 dep_no_proxy: localhost,127.0.0.1 ...
Self signed certificates
Create a private key
openssl genrsa -out yourdomain.com.key 2048
Never share generated private key
Create Certificate Signing Request (CSR)
openssl req -new -sha256 -key yourdomain.com.key -subj "/C=NL/ST=Nord Holland/L=Amsterdam/O=Surfly BV Client/OU=Dev/CN=yourdomain.com/emailAddress=support@yourdomain.com" -out yourdomain.com.csr
Sign the certificate
Generated CSR is valid only for yourdomain.com which is not enough for Surfly. When signing the certificate, please include the following additional subject identities:
subjectAltName=DNS:yourdomain.com,DNS:*.yourdomain.com
Combine private key, signed certificate and all intermediate and root certificates in one file as described here
You can use
cat
command to combine multiple files in one:cat yourdomain.com.key yourdomain.com.crt > yourdomain.com.pem
Copy your root CA certificate to
/etc/pki/ca-trust/source/anchors/
Update system certificates
sudo update-ca-trust extract
Modify
~/install/config.yaml
and add a command to append your root CA certificate to the certificate file which is used by Surfly to verify SSL connections
post_commands:
- sudo tee -a /opt/surfly/ats_certs.pem < path_to_your_root_certificate
Free certificates
You can generate free wildcard certificates with Let’s Encrypt and use them with your Surfly installation. These certificates can be generated using lego, the Let’s Encrypt binary client written in GoLang.
-
cd /tmp wget https://github.com/go-acme/lego/releases/download/v4.6.0/lego_v4.6.0_linux_amd64.tar.gz tar -xvf lego_v4.6.0_linux_amd64.tar.gz mv lego /usr/local/bin/lego
For the next step, you will need to check the lego documentation about DNS Providers to check exactly how to run the command depending on the provider used. For this example, I’ll use the Constellix documentation.
Use the following command to verify that TXT record was deployed
sudo dnf install bind-utils dig -t txt _acme-challenge.yourdomain.com
Generate the certificate
cd ~/ export DOMAIN=yourdomain.com export CONSTELLIX_API_KEY=key export CONSTELLIX_SECRET_KEY=secret lego -a --domains=${DOMAIN} --domains=*.${DOMAIN} --email=admin@mail.com -k rsa2048 --dns constellix --pem run
Generated certificates are located in
~/.lego/certificates
folder. You can check their information with:lego list
Configure Surfly installation to use the certificates. Add
ssl_certs_dir
configuration option to~/install/config.yaml
, for example... ssl_certs_dir: ["/home/client/.lego/certificates/"]
Run
~/install/setup
script to apply changes
Let’s Encrypt certificates must be renewed every 90 days. To renew a certificate using lego
it’s simple. Run the command from step 3., but with renew
instead of run
at the end. Then procceed with the steps 4. and 5.
Configuration
Surfly supports many configuration options which you can set in ~/install/config.yaml
file.
config.yaml file should have at least 3 fields:
client_id: abc
client_secret: a1b4
version: prod
config.yaml
file uses indentation to indicate the code blocks. Please pay extra attention to the section like environment
, email
and others
Do not copy configuration settings listed below directly in your config.yaml
file. Many options contain non-default values and may make your installation unusable
Surfly configuration settings
# `client_id` is the unique identifier of a client in our build system
client_id: abc
# `client_secret` together with `client_id` are used to authenticate a server in our build system
client_secret: a1b4
# `version` is used to specify the version of the build. Recommended value is `prod`
version: prod
# List of paths to the certificates (deprecated since v3.304)
# certificates: ["/home/client/mycert1.pem", "/home/client/mycert1.pem"]
# Directory with certificates in *.pem format
ssl_certs_dir: /certs
# List of the steps to skip during the installation process
skip: [
"prepare_machine", # do not update dependencies
"installation", # download a build, but skip installation process
"backup_database", # do not create a database backup copy
"backup_services", # do not create a services backup copy
"backup", # do not create any backup
]
# Use verbose output for the ansible playbook
verbose: false
# Use local build file instead of requesting it from the Surfly build server
build_file: /home/user/abc_def_ghi.xfc.tar.gz
# All variables from the section will be set as environmental variables
# on the server when running ./setup command
environment:
# Size of Varnish cache (default: 1.5G)
varnish_size: 2G
# Minimum number of threads in varnish per worker (default: 100, number of workers: 2)
varnish_min_threads: 200
# Size of short-lived Varnish cache (default: 1G)
varnish_transient_size: 1000M
# varnish storage backend. Usually "malloc" or "file,<filepath>". Default: "malloc"
varnish_storage_backend: malloc
# Size of Redis cache (default: 1G)
redis_size: 1000M
# Redis host
redis_host: 127.0.0.1
# Redis port
redis_port: 6379
# Size of in memory Apache Traffic Server cache (default: 512M)
ats_size: 512M
# Connection string to PostgreSQL database
# Examples:
# - Connect over Unix socket:
# surfly =
#
# - Remote database:
# surfly = host=1.2.3.4 port=5432 user=surfly_app password=secret
# Documentation: http://www.pgbouncer.org/config.html
db_connect_string: surfly =
# Allow Surfly to create a session on private resources. Note: if you enable
# this option, make sure your firewall configuration is strict.
# Can also be a space separated list of private IP addresses and subnets to be
# allowed
allow_private_resources: true
# Space separated list of the DNS servers (if it is not specified, it is
# populated from the /etc/resolv.conf file)
nginx_dns_resolvers: 8.8.8.8 8.8.4.4
# max number of connections per nginx process (both frontend and backend)
nginx_max_connections: 1024
# The number of processes serving proxy requests. The recommended value is
# less or equal to the number of CPU cores
proxy_workers: 4
# The number of processes serving dashboard requests. The recommended value is
# less or equal to the number of CPU cores and less than 20
dashboard_workers: 4
# The number of processes serving Session API requests
api_app_workers: 2
# Rate limit Surfly REST API requests, req/min per IP address
rest_api_rate_limit: 100
# Proxy settings
dep_http_proxy: http://10.0.0.11:8000
dep_https_proxy: http://10.0.0.11:8000
dep_ftp_proxy: http://10.0.0.11:8000
dep_no_proxy: localhost,127.0.0.1
# Default localization for new users
# The time zone has to specified as a name of an entry in the tz database
# see https://www.iana.org/time-zones and https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
default_time_zone: UTC
# The language has to be specified as a two letter iso 639-1 language code
default_language: en
# Include to set default login page to Single Sign On instead of password based login
default_login_sso: true
# Specify allowed HTTP methods for proxy application (default: all allowed (when absent))
valid_http_methods: GET POST HEAD
# Authentication token to call session API (default: randomly generated token (when absent))
# Note: Spaces are not allowed
cobro_auth_token: your_cobro_auth_oken
# Force podman containers to run as root user instead of rootless (default: false (when absent))
force_privileged: false
# Include to block all connections to dashboard backends. Default: ommited
disable_dashboard: true
# Error template directory path, this can be used to rebrand error response pages.
# For more info see the section on Surfly error in rebranding
error_template_dir: /home/client/error_templ
# Multi-server setup
#
# Unique cobro server id
cobro_server_id: 81
# Region of the server. The value is a valid session option for preferred region to start a session on
cobro_server_region: eu-west
# Description of the server. Used in `/v2/servers/` REST API endpoint
cobro_server_description: Server 81 in OVH
# IP addresses assigned to the server. This is a string with arbitrary format, it only affects
# the server description in the `/v2/servers/` REST API endpoint. You need to update this manually
# if the actual IP is changed.
cobro_server_ips: 78.47.179.204, 2a01:4f8:b0:a033::2
# Autorization token for internal dashboard API
dashboard_auth_token: your_auth_token
# Space separated list of dashboard URLs for a cobro server to register in
dashboard_urls: https://app.yourdomain.com
##
# Report attempts to violate the Content Security Policy to this URL
csp_report_uri: https://example.com/csp/
# Set timezone on the server
server_timezone: UTC
# Use the branding settings of the company with this ID to style the login
# page and as a default for users in companies without custom branding
# settings
default_branding_company: 1
# Specifies bind address for haproxy. More information is available here
# https://www.haproxy.com/documentation/hapee/latest/configuration/binds/syntax/
haproxy_listen_on: ::
# Specifies dashboard session cookie timeout, in seconds
session_cookie_age: 1209600
# Specifies the endpoints used for the session-recorder feature
start_recording_endpoint: https://recorder.example.com/start
end_recording_endpoint: https://recorder.example.com/end
# License key for Pdftron document editor, without it default pdf.js viewer is used
pdftron_key: PDFTRONKEY
# If specified, it enables server-side rendering mode in Pdftron document editor
pdftron_server: https://editor.example.com/
# Enable screenshot functionality on the server by providing credentials to
# S3 bucket to upload screenshots to
screenshots:
screenshot_s3_bucket: screenshot_bucket
screenshot_aws_access_key_id: ACCESSKEY
screenshot_aws_secret_access_key: SECRETKEY
# Enable async session history CSV extraction by providing credentials to
# S3 bucket to upload CSVs to (link to uploaded file will be sent via e-mail)
session_history_csv:
csv_upload_aws_bucket: csv_bucket
csv_upload_aws_access_key_id: ACCESSKEY
csv_upload_aws_secret_access_key: SECRETKEY
# Indicate the time period, in seconds, after which a session is classified as inactive and terminated as a 'zombie'
# if there has been no activity in the session during that time.
collect_zombie_period: 120
email:
# By default Surfly prints the content of the all outgoing emails to the logs.
# Use `surflyapp.email_backends.SMTPBackend` to send emails via your own
# SMTP server or `surflyapp.email_backends.MandrillBackend` to use your
# own Mandrill account
email_backend: "surflyapp.email_backends.ConsoleBackend"
# Default `from` email address
default_from_email: support@surfly.com
# SMTP configuration
#
# For more information, check official Django documentation
# https://docs.djangoproject.com/en/1.11/topics/email/#smtp-backend
email_host: localhost
email_port: 1025
email_host_user: client
email_host_password: password
email_use_tls: true
email_use_ssl: false
# MANDRILL configuration
#
# API key for your Mandrill account
mandrill_api_key: "your_mandrill_key"
# OAuth2 configuration for admission into session
oauth:
oauth_server_name: yourdomain.com
github_client_id: CLIENTID
github_client_secret: CLIENTSECRET
# Configuration for the Vonage SMS API
sms:
vonage_com_api_key: APIKEY
vonage_com_api_secret: APISECRET
vonage_us_phone_number: PHONENUMBER
# Install and configure dns cache. dnsmasq is installed as the caching server
# (http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html) and
# NetworkManager is configured to use it
dns_cache: false
# Email address to which notifications should be send to
notify_email: admin@yourdomain.com
# The section is used by a redundant server to synchronize state with master node (main server)
master_node:
host: 192.168.33.11 # IP address of the master node
user: client # the script establishes SSH connection with this user
notify: ["success", "failure"] # Send notifications to `notify_email` with synchronization's results
# Override redirect URLs for specific domains, specify favicon and tab title of the dashboard for specific domains
domain_redirects:
surfly.com:
contact: https://www.surfly.com/contact/
dashboard_favicon: https://docs.surfly.com/installation/favicon.ico
dashboard_title: "Surfly"
docs: https://docs.surfly.com
error404: https://some-other-404-page.example.com
home: https://www.surfly.com
register: https://some-other-register-page.example.com
successful_session_start: http://help.surfly.com/en/articles/5342170-what-defines-the-successful-start-of-a-session
videochat_docs: https://docs.surfly.com/surfly-options.html#header-video
# HTTP header to use for agent IP checks configured in Company settings,
# e.g. `X-Forwarded-For`. Set this if you have a hardware
# firewall in front of Surfly. Only the first comma-separated IP is considered
# when reading the header's value. If not set, the actual source IP is used.
client_ip_header: X-Forwarded-For
# Custom urls to verify the server status. By default https://app.your_domain.com/healthcheck/ is used
# Please make sure the domain name used here is being resolved to the server being configured
# If you want to simplify this on test environments you can also use http://localhost:8017/healthcheck/
# But this is strongly unadvised in production environments, as it won't check the certificate being used for the
# domain used on the server, and it might result in false positives during the deploy
healthcheck_urls:
- https://eu.surfly.com/healthcheck/
# Run commands after Surfly installation was successfully finished. A command might be formatted using
# values from config.yaml file:
post_commands:
- logger "Version {config[version]} is deployed"
- curl -s http://localhost:8003/CobroVersion | echo $(</dev/stdin)
# Lock a specific version of Surfly
version_lock: v3.100
# Console settings
# Email backend for Console
# base.email_backends.SMTPBackend - send emails via your own SMTP server
# base.email_backends.MandrillBackend - use your own Mandrill account
# base.email_backends.ConsoleBackend - print the content of the all outgoing emails to the logs
console_email_backend: "base.email_backends.ConsoleBackend"
# The remaining email settings are identical to those outlined in the 'email' section above.
# S3 credentials for storing Console UI assets (user avatars, company logos, etc.)
console_ui_assets_s3_access_key_id: "s3_access_key_id"
console_ui_assets_s3_secret_access_key: "s3_secret_access_key"
console_ui_assets_s3_endpoint_url: "s3_endpoint_url"
console_ui_assets_s3_bucket_name: "s3_bucket_name"
console_ui_assets_s3_region_name: "s3_region_name"
Usage
After the first installation is finished, you will be asked to create a new reseller account which you can use to log in to Surfly Dashboard. Created account will also have permissions to view License information
You can create new companies and invite agents via Surfly Dashboard or via REST API
It is recommended to have one reseller account on the server and create new clients (companies) via Dashboard interface or via REST API. If you need to create more reseller accounts please check Managing accounts section
Surfly on-premise installation does not include some of the functionality that is found on https://app.surfly.com website. For example videochat is disabled because it uses a 3rdparty service (not included in price). Please contact support@surfly.com
for details
Please check our Documentation for more details on how to use and integrate Surfly
Email configuration
Surfly does not require you to configure email functionality to create a session. However, user activation process includes an activation link from the email
Contact support@surfly.com
if you want to brand your emails
Mandrill
Edit ~/install/config.yaml
file
client_id: abc
client_secret: your_client_secret
vaersion: prod
email:
email_backend: "surflyapp.email_backends.MandrillBackend"
mandrill_api_key: "abc123"
Mandrill requires you to verify the domain name of the server by receiving an email. You can set up MX record in your DNS configuration to use your main email server for receiving emails which were sent to the domain name used by Surfly:
cobrowsing.yourdomain.com. MX 10 yourdomain.com.
Smtp server
Edit ~/install/config.yaml
file
client_id: abc
client_secret: your_client_secret
vaersion: prod
email:
email_backend: "surflyapp.email_backends.SMTPBackend"
email_host: youremailserver.com
email_port: 1025
email_host_user: client
email_host_password: password
email_use_tls: true
email_use_ssl: false
email_use_tls
and email_use_ssl
are mutually exclusive. See https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-EMAIL_USE_TLS
Reinstall Surfly server
Create backups of essential data
Create a database backup
pg_dump -U surfly_app surfly | pigz > ~/$(date +"%Y-%m-%d").gz
Copy
~/install/config.yaml
fileDepending on your setup you might also want to copy the certificates
Upgrade process
Install a new Surfly server by following instructions from here
Please check system requirements
Before running
./setup
command, copyconfig.yaml
file to~/install/
folderRestore database using the following command:
./setup restore_database -f <database backup file>
Managing accounts
There are two types of accounts: a reseller account and a company account. It is recommended to create one reseller account (normally it happens during the installation) on the server and use it to create company accounts
Creating a reseller account
./setup create_user
Registration is disabled on the server
Set a session invitation type for a company
./setup manage_command set_invite_by {COMPANY_ID} {email|sms}
Example:
./setup manage_command set_invite_by 67145 sms
Rebranding
Surfly error rebranding
Error | Description | Filename |
---|---|---|
method not allowed | This occurs when the request method is not a valid method as specified in valid_http_methods configuration option | method_not_allowed.html |
service unavailable | This error occurs when the proxy server is not available to take any request | service_unavailable.html |
session not found | This error occurs when the session doesn’t exist | session_not_found.html |
Steps to rebrand the errors:
- Create the error templates that you want to rebrand
- Put all the error templates that you made in step 1 in single directory. For example the directory is /home/client/error_templ
- Set the absolute path of the above-mentioned directory in error_template_dir under environment in configuration option.
If a custom error template is not present then the default error template will be used.
Server update
To update the server run ./setup
script from install directory. By default the setup script requests a personalized build of the latest available version of Surfly application
Every time ./setup
command is executed, a backup copy of the database is created and current build version is saved
We recommend to update Surfly weekly
Backup and restore
Backup
By default ./setup
script creates a backup every time before you install a new version. It is also possible to create a backup with the following command:
./setup backup
backup
command creates a dump of the database and archives all services.
You can find all created backups in ~/install/backupv2/
folder
Remove old backup folders to save disk space
Restore
Restoring previous Surfly version is a two-step process. First, you need to list all available backups and copy version you want to restore. Second, pass the backup version to the restore
command
List available backups
To list all available backups, run ./setup list_backups
command from the ~/install
directory
Output sample:
Backup id Surfly version Db size, MB Services, MB
2020-05-14.3 0efccd7d9ca0d52daec50ef3634c7767399f2451 88.9 KiB 70.9 MiB
list_backups
command sorts output by the modification date. The most recent backup is shown last
Restore Surfly from the backup
To start restoring Surfly from the created backup, run restore
command following by the version of the backup, for example:
./setup restore 2018-03-23.1
The command stops all running services, restores the database, installs required dependencies and restores Surfly services
Database
Surfly uses PostgreSQL as a database
By default Surfly creates a database backup every time you run ./setup
command
You might need to stop/restart the following services before restoring the database:
systemctl --user stop ss-pgbouncer
sudo systemctl restart postgresql
After the database has been manually restored, start all services:
systemctl --user start ss-surfly.target
Backup database
pg_dump -U surfly_app surfly > ~/$(date +"%Y-%m-%d").sql
Restore database
dropdb -U postgres surfly && createdb -U postgres -O surfly_app -E UTF8 surfly && psql -U surfly_app surfly < ~/2018-03-16.sql
If the database file is too big, you can use pigz
to create a compressed file:
Backup and compress database
pg_dump -U surfly_app surfly | pigz > ~/$(date +"%Y-%m-%d").gz
Restore database from a compressed file
dropdb -U postgres surfly && createdb -U postgres -O surfly_app -E UTF8 surfly && gunzip -c ~/2018-03-16.gz | psql -U surfly_app surfly
Upgrade PostgreSQL
~/install/upgrade_pg
Cache
Clear varnish cache
podman exec ss-varnish varnishadm -n /opt/varnish/ "ban req.url ~ ."
For older Surfly versions, you might need to run this command on the host instead:
varnishadm -n /opt/varnish/ "ban req.url ~ ."
Clear ATS cache
systemctl --user restart ss-trafficserver
Managing services
Surfly uses systemd for service management
Check status of all services
systemctl --user list-dependencies ss-surfly.target
Check status of a single service and view most recent logs
systemctl --user status ss-paws
Restart all services
systemctl --user restart ss-surfly.target
Stop all services
systemctl --user stop ss-surfly.target
Start all services
systemctl --user start ss-surfly.target
Failover
To provide failover capability for your Surfly setup, you will need at least 2 servers running the same version of Surfly application and a load balancer to handle failover.
To use a redundant server you need to register a new client_id
Load balancer
Surfly doesn’t require any specific load balancer to make failover work. Please check the configuration example for HAProxy:
global
log 127.0.0.1 local0 debug
user haproxy
group haproxy
defaults
log global
mode tcp
option tcplog
maxconn 10000
timeout connect 5s
timeout queue 5s
timeout client 40s
timeout server 120s
# Make statistic available on /stat (Optional)
listen stats
bind 192.168.33.10:1936
mode http
log global
maxconn 10
stats enable
stats refresh 5s
stats show-node
stats auth user:password
stats uri /stat
frontend lb_http
bind :::80 v4v6
mode http
redirect scheme https if !{ ssl_fc }
frontend lb
bind :::443 v4v6
default_backend surfly_servers
backend surfly_servers
fullconn 10000
mode tcp
server main 192.168.33.11:4433 check port 443 on-marked-up shutdown-backup-sessions send-proxy-v2
server failover 192.168.33.12:4433 check port 443 backup send-proxy-v2
When main
server 192.168.33.11 is down, all traffic is switched to the backup server 192.168.33.12.
Make sure that only the load balancer is able to establish connection on port 4433 with servers
Synchronizing Surfly version between main and redundant servers
We provide ~/install/sync_node.py
script that synchronizes both the database and Surfly version from the main server to the redundant server
Set up automatic synchronization via cron job. Type crontab -e
to create a cron job, for example:
SHELL=/bin/bash
PYTHONIOENCODING=utf8
PATH=/bin:/usr/bin:/usr/local/bin
0 3 * * * /home/client/install/sync_node_cron &>> /home/client/install/log/cron.log
To configure the synchronization process from main server to a redundant server, open ~/install/config.yaml
file and add the following section:
master_node:
host: 192.168.33.11
user: client
Main server should be able to authenticate a redundant server via SSH key
You can configure ./sync_node.py
script to verify the installation and send notifications to your email. Check notify_email option and master_node sections on Configuration page
Troubleshooting
You can contact Surfly by email support@surfly.com or call +31202611820
Changed Github RSA Key
For old installations that still used SSH keys to clone the install
repository, its possible that you will run into an issue related to Remove Host Identification when the repository tries to update itself. The error will look like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Host key for github.com has changed and you have requested strict checking.
Host key verification failed.
This is related to a change Github made to update one of its private keys. You can find more information on their blog post: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/
Their suggestion to fix this, you can simply remove the old key by running:
ssh-keygen -R github.com
After this, the next Surfly setup command should work normally.
Its also possible to make sure you have the current active keys after removing the old ones by running:
ssh-keygen -R github.com
curl -L https://api.github.com/meta | jq -r '.ssh_keys | .[]' | sed -e 's/^/github.com /' >> ~/.ssh/known_hosts
Surfly logs
Most of the Surfly logs end up in journalctl
. You can follow them by running sudo journalctl -f
.
Apache Traffic Server (ATS) logs are located in /opt/ts/var/log/trafficserver/
Ansible logs
Surfly uses Ansible to provision the server. In case of the provision script failure, it is possible to enable verbose logging by adjusting ~/install/config.yaml
file:
...
verbose: true
...