The following process should be carried out when an incident is detected with PlanX, such as a breach or DDoS attack.

Not all steps will be relevant to each situation - please use your best judgement, and act with an abundance of caution.

Keep the team, stakeholders, and users informed about the situation. We prefer open communication in shared Slack channels wherever possible.

Process

  1. Activate DDoS Protection

    Enable DDoS protection on Cloudflare to help mitigate the attack and to ensure that PlanX remains accessible to legitimate users.

  2. Monitor Traffic

    Monitor traffic on Cloudflare and server performance on CloudWatch. Try to identify any unusual patterns or spikes in traffic that could indicate an attack.

  3. Implement additional WAF Rules

    Using Cloudflare, attempt to update WAF rules to block or throttle suspicious traffic that might be part of the attack.

  4. Check CloudWatch Logs

    Examine server logs on CloudWatch to gather more information. This could help to understand the nature of the attack and the vulnerabilities being exploited.

  5. Scale Resources

    If not already underway, consider manually scaling up the server resources temporarily to handle the increased traffic load. This might help mitigate the effects of the attack.

  6. Change Credentials

    If there's any indication that the attack might have compromised PlanX’s security, change passwords, API keys, and other sensitive credentials.

  7. Backup Data

    Regularly back up our production RDS databases. Snapshots can be manually triggered via the AWS console. If the attack causes data loss or corruption, we can then restore the site from a recent backup.

  8. Report

    If the attack is malicious and you believe it's a criminal act, consider reporting it to the appropriate authorities.

  9. Resolution and Recovery

    Once the incident is under control, the team works to restore normal operations.

  10. Documentation

    All incident details, actions taken, and outcomes should be thoroughly documented. A timeline of events should be compiled for later review.

  11. Review and Improvement

    After the incident is resolved, the team conducts a post-mortem to identify lessons learned and potential improvements. Where appropriate, we will invite a facilitator from outside the development team to this meeting in order to foster clear and open communication.