Troubleshooting¶

This guide covers common issues you may encounter when deploying and operating the API Platform Event Gateway, along with their solutions.

Connection Issues¶

Gateway Not Connecting to Control Plane¶

Symptoms:

Gateway status shows as "Disconnected" in API Platform Console
Logs show connection timeout or refused errors

Solutions:

Verify network connectivity:

# Test connectivity to API Platform endpoints
curl -v https://gateway.bijira.dev/health
curl -v https://config.bijira.dev/health

Check firewall rules:

Ensure outbound HTTPS (443) is allowed to: - gateway.bijira.dev - config.bijira.dev - telemetry.bijira.dev

Verify DNS resolution:

nslookup gateway.bijira.dev
dig gateway.bijira.dev

Check proxy settings:

If using a proxy, configure the gateway:

proxy:
  http: http://proxy.example.com:8080
  https: http://proxy.example.com:8080
  noProxy: "localhost,127.0.0.1,.internal.example.com"

Intermittent Disconnections¶

Symptoms:

Gateway alternates between Connected and Disconnected states
Logs show reconnection attempts

Solutions:

Check network stability:

# Monitor connectivity over time
ping -c 100 gateway.bijira.dev

Review keep-alive settings:

connection:
  keepAliveInterval: 30s
  keepAliveTimeout: 60s
  reconnectBackoff:
    initial: 1s
    max: 60s

Check for network timeouts:

Ensure load balancers or firewalls don't have aggressive idle timeouts.

Registration Issues¶

Invalid Gateway Token¶

Symptoms:

Logs show "authentication failed" or "invalid token" errors
Gateway cannot register with control plane

Solutions:

Verify token is correct:

Copy the token again from API Platform Console without any extra spaces or characters.

Check token hasn't expired:

Tokens may have an expiration. Generate a new token if needed.

Ensure token is properly set:

Kubernetes:

kubectl get secret bijira-event-gateway-token -n bijira-event-gateway -o jsonpath='{.data.token}' | base64 -d

Docker:

docker inspect bijira-event-gateway | grep GATEWAY_REGISTRATION_TOKEN

VM:

cat configs/keys.env | grep GATEWAY_REGISTRATION_TOKEN

Token Revoked¶

Symptoms:

Previously working gateway stops connecting
Logs show "token revoked" errors

Solutions:

Generate a new token in API Platform Console.
Update the gateway configuration.
Restart the gateway.

Startup Issues¶

Gateway Pod/Container Not Starting¶

Symptoms:

Pod stuck in CrashLoopBackOff (Kubernetes)
Container exits immediately (Docker)
Service fails to start (VM)

Solutions:

Check logs for errors:

Kubernetes:

kubectl logs -n bijira-event-gateway -l app=bijira-event-gateway --previous
kubectl describe pod -n bijira-event-gateway -l app=bijira-event-gateway

Docker:

docker logs bijira-event-gateway

VM:

journalctl -u bijira-event-gateway -n 100 --no-pager

Verify resource availability:

# Kubernetes
kubectl top nodes
kubectl describe node <node-name>

# VM
free -m
df -h

Check configuration syntax:

# Validate configuration
bijira-event-gateway --config /etc/bijira/gateway.yaml --validate

Port Already in Use¶

Symptoms:

Error: "address already in use" or "port binding failed"

Solutions:

Identify the process using the port:

# Check what's using port 443
sudo ss -tlnp | grep :443
sudo lsof -i :443

Stop the conflicting process or change the port:

server:
  https:
    port: 8443  # Use alternative port

TLS/Certificate Issues¶

Certificate Not Valid¶

Symptoms:

SSL/TLS handshake failures
Browser shows certificate errors
Logs show "certificate verify failed"

Solutions:

Verify certificate validity:

openssl x509 -in /path/to/cert.crt -text -noout
openssl verify /path/to/cert.crt

Check certificate chain:

openssl s_client -connect localhost:443 -showcerts

Ensure certificate matches private key:

# Compare MD5 hashes (should match)
openssl x509 -noout -modulus -in cert.crt | openssl md5
openssl rsa -noout -modulus -in private.key | openssl md5

Certificate Expired¶

Symptoms:

SSL errors indicating expired certificate
API consumers receive TLS errors

Solutions:

Obtain a new certificate.
Update the certificate files.
Reload or restart the gateway:

# Kubernetes
kubectl rollout restart deployment/bijira-event-gateway -n bijira-event-gateway

# Docker
docker restart bijira-event-gateway

# VM
sudo systemctl reload bijira-event-gateway

Policy Issues¶

Policies Not Applied¶

Symptoms:

Policies configured in console not enforced by gateway
Policy sync status shows errors

Solutions:

Check policy sync status:

curl http://localhost:9090/health/policies

Force policy resync:

In API Platform Console, go to Settings > Event Gateways > Sync Policies.

Review gateway logs for sync errors:

# Filter for policy-related logs
grep -i "policy" /var/log/bijira/event-gateway.log

Policy Conflicts¶

Symptoms:

Unexpected behavior when multiple policies apply
Some requests allowed that should be denied (or vice versa)

Solutions:

Review policy precedence rules:
API-level policies override gateway-level policies
Explicit deny overrides allow
Check for overlapping scopes in policy definitions.
Use the policy simulation tool in API Platform Console.

Performance Issues¶

High Latency¶

Symptoms:

API response times higher than expected
Latency increases under load

Solutions:

Check resource utilization:

# Kubernetes
kubectl top pods -n bijira-event-gateway

# Docker
docker stats bijira-event-gateway

# VM
top -p $(pgrep bijira-event-gateway)

Scale horizontally:

# Kubernetes
kubectl scale deployment/bijira-event-gateway --replicas=5 -n bijira-event-gateway

Review logging level:

Debug logging can impact performance. Set to INFO or WARN in production:

logging:
  level: INFO

Check backend connectivity:

Ensure the gateway has low-latency access to backend services.

High Memory Usage¶

Symptoms:

Memory consumption grows over time
OOMKilled pods (Kubernetes)

Solutions:

Check for memory leaks:

Monitor memory over time:

watch -n 5 "kubectl top pod -n bijira-event-gateway"

Adjust resource limits:

resources:
  limits:
    memory: 4Gi
  requests:
    memory: 1Gi

Review cache settings:

cache:
  maxSize: 100MB
  ttl: 300s

API Routing Issues¶

404 Not Found Errors¶

Symptoms:

API requests return 404 errors
APIs were working previously

Solutions:

Verify API deployment:

Check if the API is deployed to this gateway in API Platform Console.

Check configuration sync:

curl http://localhost:9090/health/config

Verify the request path matches the API configuration.

502 Bad Gateway Errors¶

Symptoms:

Backend connection failures
502 errors returned to clients

Solutions:

Verify backend is accessible from gateway:

# From inside the gateway container/pod
curl -v http://backend-service:8080/health

Check backend DNS resolution:

nslookup backend-service.namespace.svc.cluster.local

Review timeout settings:

backends:
  connectTimeout: 30s
  readTimeout: 60s

Logging and Diagnostics¶

Enable Debug Logging¶

Temporarily enable debug logging for troubleshooting:

logging:
  level: DEBUG

Warning

Debug logging can generate large volumes of logs and impact performance. Disable after troubleshooting.

Collect Diagnostic Information¶

When opening a support request, collect:

Gateway version:
```
bijira-event-gateway --version
```
Configuration (redact sensitive values):
```
cat /etc/bijira/gateway.yaml
```

Recent logs:

journalctl -u bijira-event-gateway --since "1 hour ago"

Health check output:

curl http://localhost:9090/health
curl http://localhost:9090/health/config
curl http://localhost:9090/health/policies

Resource utilization:
```
top -bn1 | head -20
free -m
df -h
```

Getting Help¶

If you cannot resolve an issue using this guide:

Check the documentation: Review the relevant deployment and configuration guides.
Search known issues: Check the API Platform status page and release notes.
Contact support: Open a support ticket with diagnostic information.

What's Next?¶

Getting Started: Quick start guide
Setting Up: Configuration guide