Guides
Start typing to search…

Troubleshooting

Scanner Operations Guide

Guide for managing and troubleshooting CyberOptix scanners.

Overview

This guide covers common operational tasks for managing CyberOptix scanners, including monitoring Docker containers, viewing logs, inspecting task queues, and troubleshooting issues.

Docker Container Management

List Running Containers

View all running containers:

docker container list

View all containers (including stopped):

docker container list -a

Expected Containers:

CONTAINER ID   IMAGE                            STATUS             PORTS                        NAMES
1c6b85d1732a   greenbone/gsa:stable             Up 37 minutes      127.0.0.1:9392->80/tcp       scanner_gsa_1
f0aba441e52a   greenbone/gvmd:stable            Up 37 minutes                                   scanner_gvmd_1
ef6739989f98   greenbone/ospd-openvas:stable    Up 37 minutes                                   scanner_ospd-openvas_1
aae36445e1ba   greenbone/notus-scanner:stable   Up 37 minutes                                   scanner_notus-scanner_1
23b9d178259d   greenbone/mqtt-broker            Up 38 minutes      127.0.0.1:1883->1883/tcp     scanner_mqtt-broker_1
b069c0365bb3   greenbone/pg-gvm:stable          Up 38 minutes                                   scanner_pg-gvm_1
3e3e2121aeb9   greenbone/redis-server           Up 38 minutes                                   scanner_redis-server_1
484ec4f40d39   mongo:latest                     Up 1 hour          127.0.0.1:27017->27017/tcp   scanner_mongoDB_1
ContainerPurpose
gsaGreenbone Security Assistant web interface
gvmdGreenbone Vulnerability Manager daemon
ospd-openvasOpenVAS scanner daemon
notus-scannerNotus vulnerability scanner
mqtt-brokerMQTT message broker
pg-gvmPostgreSQL database for GVM
redis-serverRedis cache for scanner data
mongoDBMongoDB for task queue and results

View Container Logs

View logs for a specific container:

docker container logs <CONTAINER_ID>

Example - GVM Daemon Logs:

# Get container ID
docker container list | grep gvmd

# View logs
docker container logs f0aba441e52a

Follow Logs in Real-Time:

docker container logs -f <CONTAINER_ID>

View Last 100 Lines:

docker container logs --tail 100 <CONTAINER_ID>

Restart Containers

Restart a specific container:

docker restart <CONTAINER_ID>

Restart all scanner containers:

docker compose -f /etc/optix/docker-compose-optix.yml restart

Stop and Start Containers

Stop all containers:

docker compose -f /etc/optix/docker-compose-optix.yml down

Start all containers:

docker compose -f /etc/optix/docker-compose-optix.yml up -d

MongoDB Task Queue Inspection

Connect to MongoDB

Access the MongoDB shell:

mongosh

Example Session:

user@scanner:~$ mongosh
Current Mongosh Log ID:    666c374f4e0b789d5ca26a12
Connecting to:        mongodb://127.0.0.1:27017/?directConnection=true
Using MongoDB:        7.0.9
Using Mongosh:        2.2.6

test>

Inspect Task Queue

Switch to the core database and view tasks:

// Switch to core database
use core

// Set the tasks collection
coll = db.tasks

// View all tasks
coll.find()

// Count tasks
coll.countDocuments()

// Find tasks by status
coll.find({ status: "PENDING" })
coll.find({ status: "RUNNING" })
coll.find({ status: "SUCCESS" })
coll.find({ status: "FAILED" })

// Find recent tasks (last 10)
coll.find().sort({ _id: -1 }).limit(10)

Example Task Document:

{
  "_id": ObjectId("666c37554da12a358dc5f16b"),
  "name": "Web Application Discovery CLOUD_SG:1718367643",
  "task_id": Long("5853645"),
  "status": "SUCCESS",
  "content": {
    "function": "url_inspection",
    "args": {
      "urls": {
        "url_list": ["https://hostname.domain.com/"]
      }
    }
  },
  "percent": 100,
  "url_ins_result": [
    {
      "final_location": "https://hostname.domain.com/",
      "status_code": 200,
      "data": {
        "content_type": "text/html; charset=utf-8"
      }
    }
  ]
}

Common MongoDB Queries

Find Failed Tasks:

coll.find({ status: "FAILED" })

Find Tasks by Function:

// Network discovery tasks
coll.find({ "content.function": "network_discovery" })

// URL inspection tasks
coll.find({ "content.function": "url_inspection" })

// Vulnerability scan tasks
coll.find({ "content.function": "vulnerability_scan" })

Find Tasks Created Today:

// Get today's date
let today = new Date()
today.setHours(0, 0, 0, 0)

// Find tasks created today
coll.find({ _id: { $gte: ObjectId(Math.floor(today/1000).toString(16) + "0000000000000000") }})

Delete Old Completed Tasks:

// WARNING: This permanently deletes data
// Delete tasks completed more than 30 days ago
let thirtyDaysAgo = new Date()
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30)

coll.deleteMany({
  status: "SUCCESS",
  _id: { $lt: ObjectId(Math.floor(thirtyDaysAgo/1000).toString(16) + "0000000000000000") }
})

Exit MongoDB Shell

exit

Scanner Service Management

Check Service Status

View scanner service status:

sudo systemctl status scanner-active-discovery.service
sudo systemctl status scanner-client-completed-tasks.service
sudo systemctl status scanner-client-new-tasks.service
sudo systemctl status scanner-task-manager.service
sudo systemctl status scanner-gvm.service

Example Output:

● scanner-task-manager.service - CyberOptix Scanner Task Manager
     Loaded: loaded (/lib/systemd/system/scanner-task-manager.service; enabled)
     Active: active (running) since Mon 2024-06-10 16:01:33 UTC; 3 days ago
   Main PID: 397721 (scanner-task-ma)
      Tasks: 10 (limit: 9381)
     Memory: 53.5M
        CPU: 1h 15min 54.330s

Restart Services

Restart a specific service:

sudo systemctl restart scanner-task-manager.service

Restart all scanner services:

sudo systemctl restart \
  scanner-active-discovery.service \
  scanner-client-completed-tasks.service \
  scanner-client-new-tasks.service \
  scanner-task-manager.service \
  scanner-gvm.service

View Service Logs

View logs for a specific service:

sudo journalctl -u scanner-task-manager.service

Follow Logs in Real-Time:

sudo journalctl -u scanner-task-manager.service -f

View Last 100 Lines:

sudo journalctl -u scanner-task-manager.service -n 100

View Logs Since Yesterday:

sudo journalctl -u scanner-task-manager.service --since yesterday

View Logs for All Scanner Services:

sudo journalctl -u 'scanner-*' -f

System Logs

View System Log

Monitor system log in real-time:

sudo tail -f /var/log/syslog

View last 100 lines:

sudo tail -100 /var/log/syslog

Search for scanner-related entries:

sudo grep scanner /var/log/syslog

View Docker Logs

Monitor Docker daemon logs:

sudo journalctl -u docker -f

Troubleshooting

Scanner Not Receiving Tasks

Check scanner service status:

sudo systemctl status scanner-client-new-tasks.service

Check connectivity to CyberOptix:

curl -I https://your-instance-api.cyberoptix.io/health

Review task manager logs:

sudo journalctl -u scanner-task-manager.service -n 100

GVM Container Not Starting

Check container status:

docker container list -a | grep gvmd

View container logs:

docker container logs <gvmd-container-id>

Check file permissions:

ls -la /etc/optix/.gvm/

Restart GVM containers:

docker compose -f /etc/optix/docker-compose-optix.yml restart gvmd ospd-openvas

High Memory Usage

Check container resource usage:

docker stats

Check system memory:

free -h

Check MongoDB memory usage:

docker exec <mongo-container-id> mongo --eval "db.serverStatus().mem"

Failed Scans

Check MongoDB for failed tasks:

// In mongosh
use core
db.tasks.find({ status: "FAILED" }).sort({ _id: -1 }).limit(10)

Review task manager logs:

sudo journalctl -u scanner-task-manager.service --since "1 hour ago"

Check GVM scanner status:

docker compose -f /etc/optix/docker-compose-optix.yml exec gvmd gvmd --get-scanners

Disk Space Issues

Check disk usage:

df -h

Check Docker disk usage:

docker system df

Clean up Docker resources:

# Remove stopped containers
docker container prune -f

# Remove unused images
docker image prune -a -f

# Remove unused volumes
docker volume prune -f

Clean up old MongoDB tasks:

See "Delete Old Completed Tasks" in MongoDB section above.

Performance Monitoring

Monitor Container Resources

Real-time resource usage:

docker stats

Example Output:

CONTAINER ID   NAME                     CPU %     MEM USAGE / LIMIT     NET I/O           BLOCK I/O
f0aba441e52a   scanner_gvmd_1           5.2%      512MiB / 16GiB       1.2MB / 850kB     45MB / 12MB
ef6739989f98   scanner_ospd-openvas_1   15.8%     1.5GiB / 16GiB       5.2MB / 3.1MB     120MB / 45MB

Monitor System Resources

CPU and Memory:

top

or

htop  # If installed

Network Usage:

sudo iftop  # If installed

Disk I/O:

sudo iotop  # If installed

Maintenance Tasks

Update Scanner Software

# Update package lists
sudo apt update

# Upgrade scanner package
sudo apt upgrade cyberoptix.scanner -y

# Restart services
sudo systemctl restart scanner-*.service

Update Docker Images

# Pull latest images
docker compose -f /etc/optix/docker-compose-optix.yml pull

# Restart containers with new images
docker compose -f /etc/optix/docker-compose-optix.yml up -d

Backup MongoDB Data

# Create backup directory
sudo mkdir -p /backup/mongodb

# Export MongoDB data
docker exec <mongo-container-id> mongodump --out=/tmp/mongodb-backup

# Copy backup from container
docker cp <mongo-container-id>:/tmp/mongodb-backup /backup/mongodb/$(date +%Y%m%d)

Clear Task Queue

// In mongosh - USE WITH CAUTION
use core

// Clear all completed tasks
db.tasks.deleteMany({ status: "SUCCESS" })

// Clear all failed tasks
db.tasks.deleteMany({ status: "FAILED" })

// Clear all tasks (complete reset)
db.tasks.deleteMany({})

Reference Resources

Docker CLI Cheat Sheet

Docker CLI Cheat Sheet (PDF)

Common Docker Commands:

docker ps                    # List running containers
docker ps -a                 # List all containers
docker logs <container>      # View container logs
docker exec -it <container>  # Execute command in container
docker restart <container>   # Restart container
docker stop <container>      # Stop container
docker start <container>     # Start container
docker rm <container>        # Remove container

MongoDB Shell Cheat Sheet

MongoDB Shell Cheat Sheet

Common MongoDB Commands:

show dbs                     // List databases
use <database>               // Switch database
show collections             // List collections
db.<collection>.find()       // Find documents
db.<collection>.count()      // Count documents
db.<collection>.deleteMany() // Delete documents
exit                         // Exit MongoDB shell

Additional Resources

Updated about 2 months ago