Initial commit

.gitignore

+Pipfile
+Pipfile.lock
+.vscode
+*.bak
+.vagrant
+.vault.pwd
+*.retry
+venv/
+
+.vscode/
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don’t work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# celery beat schedule file
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.venv
+venv/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+
+# Certificates
+*.pem
+*.key
+

README.md

+# Timemap - GN4-3 Latency & Jitter weathermap
+
+
+This repo contains the sources and the configuation files to set the Timemap monitoring service up.
+
+The official deployment for GN4-3 is available at https://timemap.geant.org
+
+
+Go to sections
+
+- [User guide](#User guide)
+- [Admin guide](#Admin guide)
+- [Customization guide](#Customize Timemap)
+
+
+## Contacts
+
+For information about the tool and the code, please write to gn4-3-wp6-t1-lola@lists.geant.org
+
+
+
+## User guide
+
+Timemap has a minimal interface. When the user logs into the service with her home institution credentials the 
+GÈANT topology graph appears, as shown in the following image.
+
+![Topology graph](./images/TM_landing.PNG)
+
+
+Interactions on the GUI are done by selecting an element and right-clicking on it to have the context menu pop-up.
+
+By selecting a link and opening the context menu, data visualizations on the collected metric are shown. Just drag the mouse 
+pointer on the chosen metric, a new browser tab with the related time-series will open. 
+
+Link metrics are collected using RPM SNMP probes.
+
+![Link menu](./images/TM_linkmenu.PNG)
+
+
+In the same way, by right-clicking on a node the available metric visualizations will be shown. Node metrics are collected
+using TWAMP data probes.
+
+![Node menu](./images/TM_nodemenu.PNG)
+
+
+By right-clicking on the background of the topology, a menu showing information about Timemap and the overall link to the embedded
+Grafana platform appeas. 
+
+![Background menu](./images/TM_bgmenu.PNG)
+
+
+Each metric collected by Timemap have a Grafana visualization that the user can navigate to get details.
+
+![Metric visualization](./images/TM_viz.PNG)
+
+Details on a specific time interval are visible by dragging with a mouse over a portion of the reported time range.
+
+![Zoom on selected data](./images/TM_viz.PNG)
+
+A finer control on the range shown by each visualization can be done using the time picker menu, as shown in the following image. 
+
+![Timepicker](./images/TM_timepicker.PNG)
+
+
+For more details on the actions that you can perform over the visualizations, please refer to the official [Grafana documentation](https://grafana.com/docs/grafana/latest)
+
+
+## Admin guide
+
+This section is on how to deploy Timemap central services and probes
+
+### Timemap architecture
+
+Timemap uses docker containers and Telegraf agents as building blocks.
+Docker containers are used in the central service facility to deliver a data lake based on InfluxDB, a visualization tool with Grafana and additional features, for like federated authentication.
+
+Telegraf is a common tool for telemetry data collection and transport. Timemap uses custom Python plugins to fetch data from Géant routers with SNMP and from Linux servers where PerfSonar twping tool is available.
+
+The overall Timemap architecture is shown in the following figure.
+
+
+![Timemap architecture](./images/timemap-arch.png)
+
+
+### Prerequisite
+
+Central services run as Docker containers orchestrated by a Docker-compose specification file. To run the Timemap data lake you need to install Docker and Docker-compose on you Operating System. Any Linux distribution will work.
+
+You can find details on how to install Docker and Docker-compose at the following links:
+
+* [Docker installation](https://docs.docker.com/engine/install/)
+
+* [Docker Compose installation](https://docs.docker.com/compose/install/)
+
+
+
+Central services store and process data. For a production environment we recommend running Docker containers in a VM with the following characteristics:
+
+- 4 vCPU
+- 8 GB RAM
+- 20 GB storage for the VM OS
+- Additional storage for time series data may vary according to the size of your network, planned data retention, and sampling frequency. In general 20 to 40 GB will be fine for a few years of data.
+
+
+Current setup on Géant collects data from 35 routers and about 55 links producing 3.6 MB/day with a retention of 1 year and 5 minutes sampling quantum.
+
+This sums to about 1.4 GB/year. With the plan to increase the sample rate up to 1 measurement per minute, the expected data storage consumption grows to 6.4 GB/year.
+
+
+
+### Custom Docker images building
+
+Timemap relies on a custom Docker image for delivering HTML and Javascript contents supporting Federated Authentication. In addition, an auxiliary container for initializing InfluxDB and Grafana is provided.
+
+To build these custom images run the following helper script
+
+```
+[you@host <repo location>]$sudo ./docker_images/docker_build.sh
+
+```
+
+
+### Container configuration files 
+
+Docker instances are controlled at boot time using configuration files, passing environment variables from the host to the processes inside a container.
+
+Move to the repository folder containing the docker-compose orchestration file.
+
+```
+[you@host <repo location>]$cd <repo location>/timemap_central_compose
+
+```
+
+The layout of the files in this folder is the follwing
+
+```
++--- docker-compose.yml
++--- grafana
+|   +--- .env
+|   +--- grafana.ini
++--- influxdb
+|   +--- .env
++--- timemap_proxy
+|   +--- apache2
+|   |   +--- certs
+|   |   |   +--- ca.pem
+|   |   |   +--- server.key
+|   |   |   +--- server.pem
+|   +--- shibboleth
+|   |   +--- sp-cert.pem
+|   |   +--- sp-key.pem
+|   +--- shibboleth.env
+
+```
+
+Edit the following environment definition files and adapt the values to your deployment.
+
+
+```
+[you@host <repo location>/timemap_central_compose]$nano grafana/.env
+[you@host <repo location>/timemap_central_compose]$nano influxdb/.env
+[you@host <repo location>/timemap_central_compose]$nano timemap_proxy/shibboleth.env
+```
+
+Now you need a pair of certificates for Apache HTTPS and for Shibboleth Service Provider.
+
+Copy the TLS certificate, key and CA chain files into 
+
+```
+timemap_proxy/apache2/certs/ca.pem
+timemap_proxy/apache2/certs/server.pem
+timemap_proxy/apache2/certs/server.key
+```
+
+Then copy the SP certificate and key into
+
+```
+timemap_proxy/shibboleth/sp-cert.pem
+timemap_proxy/shibboleth/sp-key.pem
+```
+
+Now we are ready to deploy the Timemap central services.
+
+
+### Start/Stop/Update containers
+
+Central services lifecycle is managened using docker-compose.
+
+To start and update the services just run
+
+```
+[you@host <repo location>/timemap_central_compose]$sudo docker-compose up -d
+```
+
+To check the service status
+
+```
+[you@host <repo location>/timemap_central_compose]$sudo docker-compose ps
+```
+
+or to check containers
+
+```
+[you@host <repo location>/timemap_central_compose]$sudo docker ps
+```
+
+
+To stop the services
+
+```
+[you@host <repo location>/timemap_central_compose]$sudo docker-compose down
+```
+
+
+To remove both the containers and remove the persistent storage locations
+
+```
+[you@host <repo location>/timemap_central_compose]$sudo docker-compose down -v
+```
+
+
+### Upload predefined Grafana dashboards
+
+When the Docker containers are ready, login into Grafana using admin credentials defined in the .env file.
+
+The default Timemap dashboards are stored in
+
+```
+<repo location>/grafana_dashboards
+```
+
+You can import them by following the Import/Export steps discussed in https://grafana.com/docs/grafana/latest/dashboards/export-import/
+
+
+## Deploy Telegraf probes
+
+Timemap probes are implemented using Telegraf. Both a VM sampling routers with SNMP and perfSONAR node sampling with Twping need
+Telegraf package and two additional custom scripts to feed Timemap with new data.
+
+
+To install Telegraf on your probe nodes see the instructions on the official documentation at the following links
+
+* [Telegraf installation](https://docs.influxdata.com/telegraf/v1.18/introduction/installation/)
+
+
+Copy the custom Python scripts under /opt. These scripts are helpers that take the output of twping and SNMPtool
+commands and parse data in JSON format, that Telegraf can parse and forward to InfluxDB.
+
+```
+[you@host <repo location>]$cp telegraf_configs/rpm_twamp_snmp.py /opt/rpm_twamp_snmp.py
+[you@host <repo location>]$cp telegraf_configs/twping_json.py /opt/twping_json.py
+```
+
+Copy Telegraf configuation template, edit and adapt to you deployment as needed.
+
+```
+[you@host <repo location>]$sudo mv /etc/telegraf/telegraf.conf /etc/telegraf/telegraf.conf.orig
+[you@host <repo location>]$sudo cp telegraf_configs/telegraf.conf /etc/telegraf/telegraf.conf
+[you@host <repo location>]$sudo nano /etc/telegraf/telegraf.conf
+```
+
+Once done test the data acquisition process works without errors
+
+```
+[you@host <repo location>]$sudo telegraf --test
+```
+
+Restart and enable the service
+
+```
+[you@host <repo location>]$sudo systemctl enable telegraf
+[you@host <repo location>]$sudo systemctl restart telegraf
+[you@host <repo location>]$sudo systemctl status telegraf
+```
+
+
+To update Telegraf to newer versions just update your system as usual with yum/apt package managers.
+
+
+## Customize Timemap
+
+
+### How to adapt landing page look and feel, Cytoscape.js
+
+The service landing page is delivered by the Apache web service that takes also care of the eduGAIN authentication. 
+
+The files that must be modified to customize the landing page and the topology graph look and feel are 
+
+- docker_images/timemap_proxy/src/index.html
+- docker_images/timemap_proxy/src/wm-functions.js
+
+respectively.
+
+The graph visualization and interaction are implemented using Cytoscape.js, a Javascript library.
+
+Refer to the [Cytoscape.js official documentation](https://js.cytoscape.org/) for details.
+
+In particular, Timemap uses the [Context Menu Extension](https://github.com/cytoscape/cytoscape.js-cxtmenu) for pop-up menus.
+
+
+
+### How to adapt the topology graph
+
+GÈANT network topology is parsed automatically from GraphViz data provided by Géant operation and then converted to a Cytoscape-compliant JSON data structure.
+
+The conversion is performed by the init_timemap Docker container at service boot time.
+
+If you want to adapt the translation script, edit the file at
+
+- docker_images/init_timemap/src/parse_dot_topology.py
+
+
+For topologies other than GÈANT, init_timemap is not strictly needed. 
+
+A simpler solution could be modify the JSON data source parsed by Cytoscape directly, mapping your network topology in
+
+- docker_images/timemap_proxy/src/latest_graph.json 
+
+
+Cytoscape JSON contains an array of dictionaries representing the nodes with related metadata, followed by another array of dictionaries with information
+about the links connecting pairs of nodes with additional metadata if needed. For details on how Cytoscape models graphs please refer to the [Cytoscape.js official documentation](https://js.cytoscape.org/) 
+
+
+### How to add Grafana dashboards
+
+To add new dashboards in Grafana, just follow the official documentation at https://grafana.com/docs/grafana/latest 
+
+
+
+

docker_images/docker_build.sh

+#! /bin/bash
+
+set -o errexit -o pipefail -o noclobber -o nounset
+
+set -x
+
+docker build -t init_timemap:latest init_timemap
+docker build -t timemap_proxy:latest timemap_proxy
+
+
+docker images
\ No newline at end of file

docker_images/timemap_proxy/.dockerignore

+certs/
+shibboleth/sp-cert.pem
+shibboleth/sp-key.pem

docker_images/timemap_proxy/Dockerfile

+FROM debian:buster-backports
+
+MAINTAINER GARR System Support <system.support@garr.it>
+
+ENV TZ Europe/Rome
+
+ENV SP_HOSTNAME timemap.geant.org
+ENV SP_CONTACT gn4-3-wp6-t1-lola@lists.geant.org
+ENV SP_ABOUT /about
+
+ENV SP_ENTITY_ID https://my.entity.id
+ENV METADATA_URL https://my.metadata.url
+
+ENV SECURE_PATH_ENABLED TRUE
+
+ENV PROXY_TO elk_kibana_1 
+
+# Install required pkgs
+RUN \
+    apt-get -qq update && \
+    apt-get -qq -y --no-install-recommends install \
+        ca-certificates ntp openssl curl && \
+    apt-get -qq -y -t buster-backports --no-install-recommends install apache2 \
+        libapache2-mod-php \
+        libapache2-mod-shib2 && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install gomplate
+RUN \
+    curl -o /usr/local/bin/gomplate -sSL https://github.com/hairyhenderson/gomplate/releases/download/v3.5.0/gomplate_linux-amd64-slim && \
+    chmod 755 /usr/local/bin/gomplate
+
+# Enable apache required modules
+RUN \
+    a2enmod proxy proxy_http ssl headers alias include negotiation rewrite 
+
+# Remove default apache configuration
+RUN \
+    rm -rf /etc/apache2/sites-enabled/*
+
+# Copy apache configuration
+ADD apache2/apache2.conf /etc/apache2/apache2.conf
+ADD apache2/sites/* /etc/apache2/sites-available/
+ADD apache2/conf/* /etc/apache2/conf-available/
+
+# Copy shibboleth configuration
+ADD shibboleth/* /etc/shibboleth/
+
+# Create Folder for discovery service
+RUN mkdir -p /etc/shibboleth-ds/
+ADD shibboleth-embedded-ds-1.2.2/* /etc/shibboleth-ds/
+RUN a2enconf shibboleth-ds.conf
+
+# Create Document Root
+RUN mkdir -p /app
+# Add tracker
+COPY apache2/tracker/track.png /app/track.png
+# Add error pages
+RUN mkdir -p /error_pages
+COPY error_pages/* /error_pages/
+RUN a2enconf error-pages.conf
+
+# For testing
+RUN mkdir -p /app/secure
+COPY apache2/secure/ /app/secure/
+
+# Copy weathermap sources
+RUN mkdir -p /app/wm
+COPY src/ /app/wm/
+
+RUN htpasswd -b -c /etc/apache2/.htpasswd wp6t1user wp6t1user
+
+# Disable other_vhost_access.log
+RUN a2disconf other-vhosts-access-log.conf
+
+# Copy bootstrap script
+ADD bootstrap.sh /bootstrap.sh
+# Make executable
+RUN chmod a+rx /bootstrap.sh
+
+ENTRYPOINT [ "/bootstrap.sh" ]
+
+

docker_images/timemap_proxy/apache2/apache2.conf

+# This is the main Apache server configuration file.  It contains the
+# configuration directives that give the server its instructions.
+# See http://httpd.apache.org/docs/2.4/ for detailed information about
+# the directives and /usr/share/doc/apache2/README.Debian about Debian specific
+# hints.
+#
+#
+# Summary of how the Apache 2 configuration works in Debian:
+# The Apache 2 web server configuration in Debian is quite different to
+# upstream's suggested way to configure the web server. This is because Debian's
+# default Apache2 installation attempts to make adding and removing modules,
+# virtual hosts, and extra configuration directives as flexible as possible, in
+# order to make automating the changes and administering the server as easy as
+# possible.
+
+# It is split into several files forming the configuration hierarchy outlined
+# below, all located in the /etc/apache2/ directory:
+#
+#       /etc/apache2/
+#       |-- apache2.conf
+#       |       `--  ports.conf
+#       |-- mods-enabled
+#       |       |-- *.load
+#       |       `-- *.conf
+#       |-- conf-enabled
+#       |       `-- *.conf
+#       `-- sites-enabled
+#               `-- *.conf
+#
+#
+# * apache2.conf is the main configuration file (this file). It puts the pieces
+#   together by including all remaining configuration files when starting up the
+#   web server.
+#
+# * ports.conf is always included from the main configuration file. It is
+#   supposed to determine listening ports for incoming connections which can be
+#   customized anytime.
+#
+# * Configuration files in the mods-enabled/, conf-enabled/ and sites-enabled/
+#   directories contain particular configuration snippets which manage modules,
+#   global configuration fragments, or virtual host configurations,
+#   respectively.
+#
+#   They are activated by symlinking available configuration files from their
+#   respective *-available/ counterparts. These should be managed by using our
+#   helpers a2enmod/a2dismod, a2ensite/a2dissite and a2enconf/a2disconf. See
+#   their respective man pages for detailed information.
+#
+# * The binary is called apache2. Due to the use of environment variables, in
+#   the default configuration, apache2 needs to be started/stopped with
+#   /etc/init.d/apache2 or apache2ctl. Calling /usr/bin/apache2 directly will not
+#   work with the default configuration.
+
+
+# Global configuration
+#
+
+#
+# ServerRoot: The top of the directory tree under which the server's
+# configuration, error, and log files are kept.
+#
+# NOTE!  If you intend to place this on an NFS (or otherwise network)
+# mounted filesystem then please read the Mutex documentation (available
+# at <URL:http://httpd.apache.org/docs/2.4/mod/core.html#mutex>);
+# you will save yourself a lot of trouble.
+#
+# Do NOT add a slash at the end of the directory path.
+#
+#ServerRoot "/etc/apache2"
+
+#
+# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
+#
+#Mutex file:${APACHE_LOCK_DIR} default
+
+#
+# The directory where shm and other runtime files will be stored.
+#
+
+DefaultRuntimeDir ${APACHE_RUN_DIR}