Infrastructure
System Architecture Deployment Overview
1. ASSUMPTIONS & CLARIFICATIONS
The general knowledge that the system architecture should be designed based on:
| Component | Notes | Prerequisites | Comments |
|---|---|---|---|
| SSO | Developer Keycloak (OpenID are supported) | AITable should provide the supported protocols (OpenID preferred) | AITable support OpenID. |
| Load Balancer | F5 will be used to load balance the application services | AITable must provide the details of the exposed services by the cluster and the type of load balancing | F5 proxy to Openresty Service deployed in Kubernates by AITable, externally exposed ports: 80 and 443 |
| SSL Certificate | It's not possible to share, Developer team will configure it at the time of installation | AITable must provide the SSL certificate format | Developeruld provide a valid certificate issued by a CA, usually a digital certificate that complies with the X.509 standard. |
| K8S Cluster | Developer will provide "kubeconfig" file after the installation | AITable should provide any specific requirements for the K8S cluster (Installation guide for CentOS 7.6+) | k8s cluster version v1.22.0+ |
| K8S Stroage class | Kubernetes Cluster Storage Class for redis,minio and rabbitmq | Developer will provide Kubernetes Cluster Storage Class before the installation. Examples include: CBS, NFS... | Developer should create available Kubernetes cluster storage class and provide storage class name |
| S3 Storage/Minio OSS | Developer will provide S3 Bucket at the time of installation (But it's limited - Only App -> S3) OR S3 be part of the application but we need to define the backup strategy | AITable should prepare the application configuration to read the variables in this format: AWS_REGION='TBD', AWS_BUCKET='assets', AWS_ACCESS_KEY='TBD', AWS_ENDPOINT='TBD', AWS_ACCESS_KEY='TBD', AWS_ACCESS_SECRET='TBD' | Developer should create a bucket named [assets], with permission policy: public read and private write |
| Docker Registry | https://ghcr.io(Basic Authentication) | Developer should create the repository account and password that has permission to push & read the image | |
| MySQL Database | Developer provide MySQL databases | AITable should provide the required databases details | Developer should create a mysql account, initial database apitable with chartset utf8mb4 and configure the following parameters: max_allowed_packet = 1024M, sql_mode = NO_ENGINE_SUBSTITUTION, sort_buffer_size = 2M, max_connections=1000 |
| Terminal Operation Server | Used for AITable Application Installation and Operation | Developer will provide the information at the time of installation | Terminal Operation Server needs to be able to connect to the k8s cluster |
| AITable Web Server | Build supercharged, SEO-friendly, and extremely user-facing static website and web application by using NextJS | K8S Cluster & Docker Registry & S3 Storage & SSL Certificate & MySQL Database & Terminal Operation Server | |
| AITable Backend Server | Handles HTTP requests about nodes, users, organizations, etc. Code under Java Spring Boot Framework | K8S Cluster & Docker Registry & S3 Storage & SSL Certificate & MySQL Database & Terminal Operation Server | |
| AITable Room Server | Used for operations of datasheets, communicates with Socket Server through gRPC. nest-server: handles HTTP GET requests about datasheets, records, views, etc. socket-server: establishes a long connection with clients through the WebSocket protocol, allowing for two-way communication and real-time collaboration, notifications, and other features | K8S Cluster & Docker Registry & S3 Storage & SSL Certificate & MySQL Database & Terminal Operation Server | |
| Redis | Stores cache, such as log-in session, hot data, etc. | K8S Cluster & Docker Registry & S3 Storage & SSL Certificate & MySQL Database & Terminal Operation Server | Redis uses master-slave mode and needs to enable password authentication |
| RabbitMQ | Message broker software that acts as an intermediary for messaging between different applications or components. | K8S Cluster & Docker Registry & S3 Storage & SSL Certificate & MySQL Database & Terminal Operation Server | RabbitMQ uses master-slave mode and needs to enable password authentication |
2. APPLICATION COMPONENTS
List of the project's different application components and each of these component's role in the system architecture:
| # | Component | Purpose | Framework | Version | Interactions |
|---|---|---|---|---|---|
| 1 | Web Server | Serving the frontend app to users and as a reverse proxy | NextJS | [Version 1.10.0] | Forwards requests to appropriate backend services |
| 2 | Frontend Server | Handling user interface and interactions | NextJS | [Version 1.10.0] | Communicates with Backend Server and Room Server |
| 3 | Backend Server | Managing business logic and data processing | Spring Boot | [Version 1.10.0] | Interacts with Database, Redis, and S3 Storage |
| 4 | Room Server (nest-server) | Handling HTTP GET requests for datasheets, records, views, etc. | NestJS | [Version 1.10.0] | Communicates with Database and Redis |
| 5 | Room Server (socket-server) | Establishing WebSocket connections for real-time features | NextJS | [Version 1.10.0] | Maintains real-time connections with clients, interacts with Redis |
| 6 | Imageproxy Server | Image cropping service | Golang | [Version v0.13.4] | Interacts with S3 Storage |
| 7 | Database | Storing application data | MySQL | [Version 8.0.32+] | Accessed by Backend Server and Room Server |
| 8 | Cache | Storing session and hot data | Redis | [Version 6.0+] | Used by Backend Server and Room Server for quick data access |
| 9 | Object Storage | Storing files and objects | S3-compatible | [Version latest] | Accessed by Backend Server for file operations |
| 10 | RabbitMQ | Message queue processing middleware | S3-compatible | [Version 3.13.0+] | Accessed by Backend Server for file operations |
3. SYSTEM ARCHITECTURE
4. SYSTEM RESOURCE ALLOCATION
4.1 Staging
| System | Quantity | Resources | OS | Functionality | Requirements |
|---|---|---|---|---|---|
| Kubernetes Control Plane (Master) | 1 | [4] cores vCPU, [8]GB RAM, [100]GB Storage Space | CentOS 7.6+ | Responsible for managing the cluster | Kubernetes API |
| Kubernetes Node (Worker) | 2 | [8] cores vCPU, [16]GB RAM, [100]GB Storage Space | CentOS 7.6+ | A node is a VM that serves as a worker machine in a Kubernetes cluster, that will host the application components | 1. Kubelet, a process responsible for communication between the Kubernetes control plane and the Node; it manages the Pods and the containers running on a machine. 2. A container runtime (like Docker) responsible for pulling the container image from a registry, unpacking the container, and running the application. |
| Terminal Operation Server | 1 | 2 cores vCPU, 4GB RAM, 100GB Storage Space | CentOS 7.6+ | Used for Application Installation and Operation | kubectl |
| Database Server | 1 | [8] cores vCPU, [16]GB RAM, [200]GB Storage Space | CentOS 7.6+ | The Main Database for the Application | MySQL [Version 8.0.32+] |
4.2 Production
| System | Quantity | Resources | OS | Functionality | Requirements |
|---|---|---|---|---|---|
| Kubernetes Control Plane (Master)* | 2 | [4] cores vCPU, [8]GB RAM, [100]GB Storage Space | CentOS 7.6+ | Responsible for managing the cluster | Kubernetes API |
| Kubernetes Node (Worker)* | 3 | [16] cores vCPU, [64]GB RAM, [200]GB Storage Space | CentOS 7.6+ | A node is a VM that serves as a worker machine in a Kubernetes cluster, that will host the application components | Kubelet, Docker |
| Terminal Operation Server | 1 | 2 cores vCPU, 4GB RAM, 500GB Storage Space | CentOS 7.6+ | Used for Application Installation and Operation | kubectl |
| ..... |
* Production traffic should have a minimum of three nodes because if one node goes down, both an etcd member and a control plane instance are lost, and redundancy is compromised. You can mitigate this risk by adding more control plane nodes.
Scalability Considerations
- Kubernetes Nodes: Start with the recommended configuration. Monitor resource utilization and adding more nodes as needed.
- Database: Read replicas for scaling read operations.
- S3 Storage: Chosen solution can scale to handle increasing storage needs.
5. CONFIGURATION PATH
| Component | Path | Description |
|---|---|---|
| Terminal Operation Server | /data/apitable/app/state | Stores the installation information |
| Terminal Operation Server | /data/apitable/app/config-k8s | Stores kubeconfig (Add our k8s cluster config file by the name "kubeconfig" inside the folder config-k8s) |
| Terminal Operation Server | /data/apitable/app/custom-config | Stores configuration files |
6. SYSTEM BACKUP AND DISASTER RECOVERY
| Component | Backup Description | Frequency | Retention | Restore Procedure |
|---|---|---|---|---|
| Database | Full backup + Incremental backups | Full: Weekly, Incremental: Daily | 30 days | Use MySQL backup tools to restore from the latest backup |
| Configuration Files | Version-controlled repository | On every change | Indefinite | Clone repository and apply configurations |
| Kubernetes State | etcd backup | Daily | 7 days | Restore etcd data from backup |
7. FIREWALL POLICY
| Service | Source | Destination | Port | Protocol |
|---|---|---|---|---|
| Load Balancer to Application | Load Balancer IP | K8S | 80/443 | TCP |
| Download the AITable | Terminal Operation Server IP | https://ghcr.io/ | 443 | TCP |
| Database Access | K8S Nodes | Database Server | 3306 | TCP |
| Redis Access | K8S Nodes | Redis Server | 6379 | TCP |
| S3 Access | K8S Nodes | S3 Storage/Minio OSS | 443 | TCP |
| Kubernetes API | K8S Nodes | K8S Control Plane | 6443 | TCP |
8. MONITORING AND LOGGING
Monitoring
| Component | Metrics to Monitor |
|---|---|
| Kubernetes Cluster | CPU, Memory, Disk Usage, Pod Status |
| Web Server | Request Rate, Error Rate, Latency |
| Database | Connections, Query Performance |
| Redis | Memory Usage, Evictions, Connections |
| S3 Storage/Minio OSS | Bucket Size, Request Count, Error Rate |
Logging
- Retain application logs for 30 days
- Retain system logs for 90 days
- Implement log rotation to manage disk space