Production SSH Access Skill

Overview

This skill guides you through accessing and working with the Codante.io API production server hosted on Digital Ocean.

Server Information

• Host: 216.238.103.47
• User: robertotcestari
• OS: Linux (Ubuntu/Debian)
• Web Server: nginx
• Application Server: Laravel Octane (RoadRunner on port 8089)
• Backend: PHP 8.1+
• Database: MySQL 8.0
• Cache/Queue: Redis
• Application Root: /var/www/codante-io-api/current (symlink to active release)
• Deployment Structure: Capistrano-style with current, releases/, and shared/ directories
• Process Manager: Supervisor (manages Octane and queue jobs)

Quick Access

Connect to Production Server

ssh [email protected]

Common SSH Commands

List nginx sites:

ssh [email protected] "ls /etc/nginx/sites-enabled/"

Check nginx configuration:

ssh [email protected] "sudo nginx -t"

Check service status:

ssh [email protected] "sudo supervisorctl status codante-octane"
ssh [email protected] "systemctl status nginx"
ssh [email protected] "systemctl status mysql"
ssh [email protected] "systemctl status redis-server"

Check Supervisor status:

ssh [email protected] "sudo supervisorctl status"

File Transfer

WARNING: do not transfer files - this is used only for emergency debugging. To change files in the production server, use the deployment pipeline.

Copy files from production to local:

scp [email protected]:/path/on/server /local/path

Copy files from local to production:

scp /local/path [email protected]:/path/on/server

Production Application Paths

• Application Current: /var/www/codante-io-api/current
• Releases: /var/www/codante-io-api/releases/ (numbered directories: 1, 2, 3, 4...)
• Shared Files: /var/www/codante-io-api/shared/ (persistent data between releases)
• Logs Directory: /var/www/codante-io-api/current/storage/logs/
• Public: /var/www/codante-io-api/current/public (served by nginx)
• .env Config: /var/www/codante-io-api/shared/.env

Log Access in Production

Codante uses multiple logging channels with all logs stored in /var/www/codante-io-api/current/storage/logs/.

Main Application Logs

Laravel Main Log:

# View live logs (follow mode)
ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/laravel.log"

# View last 100 lines
ssh [email protected] "tail -100 /var/www/codante-io-api/current/storage/logs/laravel.log"

# Get file size and info
ssh [email protected] "ls -lh /var/www/codante-io-api/current/storage/logs/laravel.log"

Specialized Log Channels

Deprecation Logs (PHP deprecation warnings):

ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/deprecations.log"

Security Logs (daily files, kept for 14 days):

ssh [email protected] "ls -lh /var/www/codante-io-api/current/storage/logs/security*.log"
ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/security-2025-10-22.log"

Payment Logs (Pagarme integration):

ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/payments.log"

Discord Logs (Discord integration):

ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/discord.log"

nginx Logs

nginx access logs:

ssh [email protected] "tail -f /var/log/nginx/access.log"

nginx error logs:

ssh [email protected] "tail -f /var/log/nginx/error.log"

Site-specific error logs:

ssh [email protected] "ls -lh /var/log/nginx/*.log"

System Logs - Services

Octane service logs:

ssh [email protected] "journalctl -u codante-octane -n 100 -f"

nginx service logs:

ssh [email protected] "journalctl -u nginx -n 100"

MySQL service logs:

ssh [email protected] "journalctl -u mysql -n 100"

Redis logs:

ssh [email protected] "journalctl -u redis-server -n 100"

Supervisor logs:

ssh [email protected] "tail -f /var/log/supervisor/supervisord.log"

View All Logs in Directory

# List all log files with sizes
ssh [email protected] "ls -lh /var/www/codante-io-api/current/storage/logs/"

# Count lines in each log file
ssh [email protected] "wc -l /var/www/codante-io-api/current/storage/logs/*"

# Get total size
ssh [email protected] "du -sh /var/www/codante-io-api/current/storage/logs/"

Search and Filter Logs

Find errors in Laravel logs:

ssh [email protected] "grep 'ERROR\|Exception' /var/www/codante-io-api/current/storage/logs/laravel.log | head -50"

Find errors from specific date:

ssh [email protected] "grep '\[2025-10-22' /var/www/codante-io-api/current/storage/logs/laravel.log | grep 'ERROR'"

Search for specific errors:

ssh [email protected] "grep 'payment\|webhook\|queue\|notification' /var/www/codante-io-api/current/storage/logs/laravel.log"

Log Format and Structure

Laravel logs use Monolog format:

[YYYY-MM-DD HH:MM:SS] CHANNEL.LEVEL: MESSAGE {"context": "data"}

Example:

[2025-10-22 14:32:15] laravel.ERROR: Exception occurred {"exception":"[object] (Exception(code: 0): User not found at /app/User.php:123)"...}

Download Logs Locally

Copy single log file:

scp [email protected]:/var/www/codante-io-api/current/storage/logs/laravel.log ./laravel-backup.log

Copy all logs:

scp -r [email protected]:/var/www/codante-io-api/current/storage/logs/ ./production-logs/

Copy security logs:

scp [email protected]:/var/www/codante-io-api/current/storage/logs/security*.log ./

Manage Large Log Files

Check log sizes:

ssh [email protected] "du -h /var/www/codante-io-api/current/storage/logs/"

Rotate/Clear logs via Artisan:

ssh [email protected] "cd /var/www/codante-io-api/current && php artisan log:clear"

Manually rotate specific log:

ssh [email protected] "cd /var/www/codante-io-api/current/storage/logs && mv laravel.log laravel.log.$(date +%Y%m%d) && gzip laravel.log.* && ls -lh"

Real-time Log Monitoring

Watch multiple logs simultaneously:

ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/laravel.log /var/log/nginx/error.log"

Monitor with grep filter:

ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/laravel.log | grep -E 'ERROR|Exception|Warning'"

Watch payment logs in real-time:

ssh [email protected] "tail -f /var/www/codante-io-api/current/storage/logs/payments.log"

Supervisor Process Management

Codante uses Supervisor to manage Laravel Octane and queue workers.

Check Supervisor Status

# Status of all programs
ssh [email protected] "sudo supervisorctl status"

# Status of Octane specifically
ssh [email protected] "sudo supervisorctl status codante-octane"

# Status of queue workers (if configured)
ssh [email protected] "sudo supervisorctl status codante-worker"

View Supervisor Logs

# Main supervisor log
ssh [email protected] "tail -f /var/log/supervisor/supervisord.log"

# Octane logs
ssh [email protected] "tail -f /var/log/supervisor/codante-octane.log"

# Queue worker logs (if configured)
ssh [email protected] "tail -f /var/log/supervisor/codante-worker.log"

Restart Supervisor Programs

# Restart Octane
ssh [email protected] "sudo supervisorctl restart codante-octane"

# Restart all programs
ssh [email protected] "sudo supervisorctl restart all"

# Start/stop Octane
ssh [email protected] "sudo supervisorctl start codante-octane"
ssh [email protected] "sudo supervisorctl stop codante-octane"

Monitor Queue Jobs

# View active queue jobs
ssh [email protected] "cd /var/www/codante-io-api/current && php artisan queue:failed"

# Monitor queue in real-time
ssh [email protected] "cd /var/www/codante-io-api/current && php artisan queue:monitor redis:default --timeout=300"

# Retry failed jobs
ssh [email protected] "cd /var/www/codante-io-api/current && php artisan queue:retry all"

Check Queue Configuration

ssh [email protected] "cat /etc/supervisor/conf.d/codante-worker.conf"

Common Production Tasks

Check Disk Space

ssh [email protected] "df -h"

Monitor CPU/Memory

ssh [email protected] "top -b -n 1 | head -20"

Check Process Memory Usage

# Octane processes
ssh [email protected] "ps aux | grep octane"

# nginx processes
ssh [email protected] "ps aux | grep nginx"

# Check memory by Octane workers
ssh [email protected] "ps aux | grep octane | awk '{print \$2, \$4, \$6, \$11}'"

Restart Services

# Restart Octane (via Supervisor)
ssh [email protected] "sudo supervisorctl restart codante-octane"

# Restart nginx
ssh [email protected] "sudo systemctl restart nginx"

# Restart MySQL
ssh [email protected] "sudo systemctl restart mysql"

# Restart Redis
ssh [email protected] "sudo systemctl restart redis-server"

# Restart Supervisor (and all managed programs)
ssh [email protected] "sudo systemctl restart supervisor"

# Restart all services
ssh [email protected] "sudo systemctl restart nginx mysql redis-server && sudo supervisorctl restart all"

Clear Cache

ssh [email protected] "cd /var/www/codante-io-api/current && php artisan cache:clear && php artisan optimize:clear"

Database Operations

# Run migrations
ssh [email protected] "cd /var/www/codante-io-api/current && php artisan migrate"

# Run seeders
ssh [email protected] "cd /var/www/codante-io-api/current && php artisan db:seed"

# Check database connection
ssh [email protected] "cd /var/www/codante-io-api/current && php artisan tinker"

Check Current Deployment

# See current release
ssh [email protected] "ls -l /var/www/codante-io-api/current"

# Check recent commits
ssh [email protected] "cd /var/www/codante-io-api/current && git log -5 --oneline"

# Check recent releases
ssh [email protected] "ls -la /var/www/codante-io-api/releases/ | tail -5"

Deploy New Release

When deploying via GitHub Actions + Deployer:

1. Merge to main branch
2. GitHub Actions triggers automated deployment
3. Deployer handles:

- Code checkout to new release directory - Dependency installation - Cache clearing - Database migrations - Asset compilation - Service restarts (PHP-FPM, nginx, Supervisor)

Check deployment status:

ssh [email protected] "ls -la /var/www/codante-io-api/releases/ | tail -5"

Health Check

# Quick API health check
ssh [email protected] "curl -s http://127.0.0.1/health"

# Detailed status
ssh [email protected] "curl -s http://127.0.0.1/api/status | head -20"

# Check if PHP-FPM is responding
ssh [email protected] "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/"

Security Considerations

1. SSH Keys: Ensure your SSH key is added to ~/.ssh/authorized_keys on the server
2. sudo Access: Use sudo for privileged commands (requires password or sudo setup)
3. Environment Variables: Never expose sensitive data in logs or console output
4. Log Files: Check logs for errors, security warnings, and performance issues
5. Backups: Always ensure backups exist before making changes to production
6. Worker Cache: Be aware that Supervisor-managed workers may cache code; restart after major changes
7. Log Retention: Security logs kept for 14 days; archive important logs locally
8. Shared Config: .env and other shared config in /var/www/codante-io-api/shared/ - changes survive deployments
9. Queue Jobs: Failed jobs are retained; monitor and clean up periodically

Troubleshooting

Connection Issues

If you can't connect:

1. Verify SSH key is configured: ssh-keyscan 216.238.103.47
2. Check your SSH config: ~/.ssh/config
3. Test connectivity: ping 216.238.103.47

Permission Denied

If you get "Permission denied":

1. Verify the username is robertotcestari
2. Check SSH key permissions: chmod 600 ~/.ssh/id_rsa
3. Ensure public key is on server: cat ~/.ssh/id_rsa.pub

Service Issues

If services are not running:

1. Check service status: sudo systemctl status SERVICE_NAME
2. View service logs: sudo journalctl -u SERVICE_NAME -n 50
3. Restart service: sudo systemctl restart SERVICE_NAME

Octane Not Responding

If Octane isn't responding:

1. Check if Octane is running: sudo supervisorctl status codante-octane
2. View recent logs: tail -f /var/log/supervisor/codante-octane.log
3. Restart Octane: sudo supervisorctl restart codante-octane
4. Check if bound to port 8089: sudo netstat -tlnp | grep 8089 or sudo ss -tlnp | grep 8089
5. Check Octane service via journalctl: journalctl -u supervisor -n 100 | grep octane

Supervisor Worker Issues

If queue workers are not processing:

1. Check supervisor status: sudo supervisorctl status
2. View worker logs: tail -f /var/log/supervisor/codante-worker.log
3. Restart workers: sudo supervisorctl restart codante-worker
4. Check for failed jobs: cd /var/www/codante-io-api/current && php artisan queue:failed

High Memory Usage

If Octane is consuming too much memory:

1. Check current memory usage: free -h
2. Check Octane memory: ps aux | grep octane | awk '{print $2, $4, $6, $11}'
3. Current config should trigger garbage collection at 50MB
4. Restart Octane: sudo supervisorctl restart codante-octane
5. Monitor with: watch -n 1 'ps aux | grep octane'
6. Review logs for memory leak patterns in /var/log/supervisor/codante-octane.log

Queue Job Issues

If queue jobs are failing or not processing:

1. Check queue status: cd /var/www/codante-io-api/current && php artisan queue:failed
2. View worker logs: tail -f /var/log/supervisor/codante-worker.log
3. Retry failed jobs: cd /var/www/codante-io-api/current && php artisan queue:retry all
4. Clear failed jobs: cd /var/www/codante-io-api/current && php artisan queue:clear
5. Monitor queue: cd /var/www/codante-io-api/current && php artisan queue:monitor redis:default

Log Files Growing Too Large

If log files are consuming too much disk space:

1. Check size: du -sh /var/www/codante-io-api/current/storage/logs/
2. Clear logs: cd /var/www/codante-io-api/current && php artisan log:clear
3. Archive old logs: gzip /var/www/codante-io-api/current/storage/logs/*.log.old
4. Download and remove: scp ... && ssh ... rm

nginx 502 Bad Gateway

If you see 502 errors in nginx:

1. Check if Octane is running: sudo supervisorctl status codante-octane
2. Check nginx error log: tail -f /var/log/nginx/error.log
3. Test Octane is bound to port 8089: curl http://127.0.0.1:8089
4. Restart Octane: sudo supervisorctl restart codante-octane
5. Check Octane logs: tail -f /var/log/supervisor/codante-octane.log

Payment Processing Issues

If Pagarme webhooks are failing:

1. Check payment logs: tail -f /var/www/codante-io-api/current/storage/logs/payments.log
2. Verify webhook configuration in .env
3. Check failed queue jobs: cd /var/www/codante-io-api/current && php artisan queue:failed
4. Retry webhook jobs: cd /var/www/codante-io-api/current && php artisan queue:retry all

When to Use This Skill

Use this skill when you need to:

• ✅ Connect to the production server
• ✅ Debug production issues
• ✅ Access and analyze production logs
• ✅ Monitor real-time log streams
• ✅ Check server logs and metrics
• ✅ Manage services and processes
• ✅ Work with Supervisor and queue jobs
• ✅ Transfer files to/from production
• ✅ Run maintenance commands
• ✅ Troubleshoot deployment issues
• ✅ Monitor performance and resource usage
• ✅ Manage Laravel queue workers
• ✅ Handle payment and webhook issues

Related Resources

• CLAUDE.md deployment section for CI/CD information
• Server management documentation
• Deployer configuration for automated deployments
• Supervisor documentation: http://supervisord.org/
• Laravel Queue documentation: https://laravel.com/docs/queues
• Laravel Commands documentation: https://laravel.com/docs/artisan