Skip to main content

Bidding

Updated System Design for a Campaign Management System​

Since the system updates campaign data once per day in the morning, we can optimize it for batch processing rather than real-time updates. Here’s an updated design focusing on the daily refresh and including a high-level architecture diagram description:

1. High-Level Design Overview​

The system will:

  • Add clients via APIs for Yandex Direct, Google Ads, Meta, VK, and MyTarget.
  • Fetch campaign data for each client and store it.
  • Perform daily updates in the morning to download all campaigns and targeting data.
  • Store historical data for reports and analysis.
  • Execute predefined rules for campaign activation, deactivation, or optimization.
  • Store rules and apply them daily based on conditions (e.g., campaign budget, time of day).

2. System Components​

  1. API Integration Layer:

    • Fetches campaign data from different advertising platforms (Yandex Direct, Google Ads, Meta, VK, and MyTarget).
    • Uses scheduled batch processing to download data once every day in the morning.

  2. Data Storage:

    • Amazon RDS (PostgreSQL): Stores client data, campaign details, targeting, and rules.
    • Amazon S3: Stores large historical reports and campaign performance data for long-term storage.

  3. Processing Layer:

    • AWS Lambda: Executes daily batch jobs for fetching campaign data and updating campaign rules.
    • AWS Step Functions: Orchestrates batch workflows, handling API rate limits and retries for campaign data retrieval.

  4. Rule Engine:

    • Lambda will also handle rule execution daily based on the downloaded data. For example, rules for enabling/disabling campaigns based on conditions like time, budget, or other factors.

  5. Data Analysis:

    • Athena/Redshift Spectrum: Analyze historical campaign data stored in S3 without moving it to RDS, enabling quick report generation.

3. Detailed Data Flow​

a. Client Addition​

  1. A user provides API credentials for a specific platform (e.g., Yandex Direct, Google Ads).
  2. The system makes an initial API call to retrieve the list of campaigns and targeting data.
  3. Store campaigns, keywords, or behavioral targets in PostgreSQL.

b. Daily Data Sync (Morning Update)​

  1. AWS Lambda is triggered every morning at a scheduled time (e.g., 6:00 AM) to initiate the daily data sync.
  2. AWS Step Functions orchestrates calls to the APIs of each ad platform to fetch updated campaign data for all 1000 clients.
  3. Campaign data is retrieved and stored in PostgreSQL, while large historical reports (up to 10 MB per campaign) are archived in Amazon S3.

c. Rule Execution​

  1. After the data sync, AWS Lambda checks for rules in the Rules Table. Rules can include conditions like:
    • Enable/disable campaigns if the budget is above or below a certain threshold.
    • Enable/disable campaigns based on the current date.
  2. Lambda updates campaign status in PostgreSQL according to these rules.

d. Historical Data Storage​

  1. Campaign data before the update is archived in Amazon S3 to maintain a history.
  2. Athena/Redshift Spectrum allows users to run SQL-like queries over historical campaign data directly in S3.

4. System Scaling Considerations​

  • Batch Processing: Since updates are only required once a day, the system can handle the API load in batch mode. It does not need to process real-time data, reducing complexity.
  • Storage: With large volumes of historical data (potentially 1 TB per year), using S3 for archival storage is cost-effective. RDS stores only active campaign data for fast access.
  • Scaling AWS Components:
    • Lambda and Step Functions scale automatically to handle the load of 1000 clients during the daily update.
    • S3 provides virtually unlimited storage for historical reports and campaign data.

5. Database Schema​

Campaign Table​

FieldTypeDescription
campaign_idUUIDUnique ID for each campaign
client_idUUIDForeign key to client
platformEnumPlatform (Yandex, Google, Meta, etc.)
keywordsJSONBList of keywords or behavioral targets
start_dateTimestampStart date of the campaign
end_dateTimestampEnd date of the campaign
statusEnumCampaign status (active, paused, etc.)
daily_budgetDecimalDaily budget for the campaign

History Table​

FieldTypeDescription
history_idUUIDUnique ID for the historical record
campaign_idUUIDForeign key to campaign
change_timestampTimestampWhen the data was archived
data_snapshotJSONBSnapshot of the campaign's data at this time

Rules Table​

FieldTypeDescription
rule_idUUIDUnique ID for each rule
conditionJSONBRule conditions (e.g., budget threshold, time)
actionJSONBAction to take when condition is met
campaign_idUUIDForeign key to campaign

6. High-Level Architecture Diagram (Textual Description)​

  • Client: Adds API credentials through the web interface.
  • API Integration Layer: The integration layer fetches campaign data for Yandex Direct, Google Ads, Meta, VK, and MyTarget. This data is sent to the storage layer.
  • Processing Layer: AWS Lambda functions handle the daily syncing of campaigns and rule execution. AWS Step Functions coordinate the API calls.
  • Data Storage:
    • RDS stores active campaign data, including keywords, budgets, and targeting information.
    • S3 archives historical campaign data and performance reports.
  • Rule Engine: Rules are stored in RDS and executed via Lambda after syncing campaigns.
  • Data Analytics: Historical campaign data stored in S3 is queried using Athena or Redshift Spectrum for reporting purposes.

7. Technologies Used:​

  • API Integration: Custom connectors to Yandex Direct, Google Ads, Meta, VK, MyTarget APIs.
  • Storage: PostgreSQL (Amazon RDS) for structured data, Amazon S3 for large historical data.
  • Processing: AWS Lambda, AWS Step Functions for batch processing and rules.
  • Querying: Amazon Athena or Redshift Spectrum for querying historical data in S3.
  • Scheduling: AWS CloudWatch for scheduling daily Lambda triggers.

Таймеры и выполнение​

Для того чтобы таймеры из правил могли достигать конечного API и выполнять операции, такие как отключение рекламных кампаний (РК), предлагается использовать комбинацию событийного подхода с функциями AWS Lambda. Вот как это может быть реализовано:

1. Правила с таймерами для РК​

Правила для управления кампаниями по времени или по условиям могут включать, например:

  • Отключение кампании в определённое время (например, 9:00 утра).
  • Отключение кампании при достижении определённого лимита бюджета.

Эти правила будут храниться в таблице правил в базе данных (RDS). Структура таблицы правил уже приведена, но дополним её для работы с таймерами.

Пример схемы для таблицы правил:​

ПолеТипОписание
rule_idUUIDУникальный ID для каждого правила
campaign_idUUIDID рекламной кампании, к которой применимо правило
action_typeStringТип действия (включение, отключение, корректировка бюджета)
conditionJSONBУсловия, такие как время выполнения или бюджет
execution_timeTimestampВремя выполнения действия

2. Механизм таймеров и выполнения правил​

a. Запуск выполнения правил по расписанию (таймеры):​

  1. AWS EventBridge (ранее Amazon CloudWatch Events) будет использоваться для создания расписания выполнения правил.

    • EventBridge позволяет настроить крон-выражения или фиксированные интервалы времени для выполнения определённых событий. Например, вы можете создать правило, которое будет запускать проверку всех правил в базе данных каждые 10 минут или в любое другое время.

  2. AWS Lambda: В EventBridge настраивается правило, которое раз в заданный интервал вызывает функцию Lambda. Lambda обращается к базе данных, извлекает все правила с истёкшим временем выполнения (текущие таймеры) и проверяет, какие действия необходимо выполнить.

b. Выполнение действия из правил:​

  1. AWS Lambda запускается по расписанию и проверяет в базе данных правила, которые нужно выполнить (например, если время execution_time правила уже наступило).
  2. Lambda извлекает все подходящие правила с условиями и действиями, которые необходимо выполнить, такие как отключение рекламных кампаний.
  3. Для выполнения действия, например, отключения РК, Lambda вызывает API соответствующей рекламной платформы (например, Google Ads API или Yandex Direct API).

c. Пример процесса отключения кампании:​

  1. Lambda запускается в 9:00 утра, проверяя все правила в таблице, где execution_time меньше текущего времени.
  2. Находит правило для кампании с campaign_id = '12345', где действие — выключить кампанию.
  3. Lambda делает запрос к API рекламной системы (например, Yandex Direct) через HTTP запрос: 
    POST https://api.direct.yandex.com/json/v5/campaigns
    Body: {
      "method": "suspend",
      "params": {
        "SelectionCriteria": {
          "Ids": ["12345"]
        }
      }
    }
  4. После успешного отключения рекламной кампании, статус обновляется в базе данных (RDS) для синхронизации с актуальным состоянием.

3. Оптимизация через асинхронное выполнение и мониторинг​

  • Асинхронное выполнение: Если в одном запуске необходимо отключить множество кампаний, система должна работать асинхронно. Каждое действие по отключению может запускаться параллельно, чтобы минимизировать задержки. Lambda может обрабатывать несколько правил одновременно.
  • Мониторинг и обработка ошибок: Для отслеживания состояния выполнения действий можно настроить мониторинг через AWS CloudWatch. Если отключение кампании по каким-то причинам не удается (например, из-за ошибки API), можно настроить повторные попытки (retry logic) через Step Functions или добавить уведомление через SNS (Simple Notification Service).

4. Архитектура выполнения правила с таймером​

  1. EventBridge (CloudWatch Events): Планирует запуск функции Lambda каждые X минут для выполнения проверки на правила.

  2. Lambda:

    • Извлекает все правила, у которых время выполнения уже наступило.
    • Выполняет действия, такие как отключение или включение кампаний, отправляя HTTP запросы к API рекламных платформ (Yandex, Google, Meta и др.).

  3. Рекламные платформы API: Принимают запросы от Lambda для выполнения операций (например, POST /campaigns/suspend для отключения кампании).

  4. База данных (RDS): Хранит данные о кампаниях, правилах и статусах. После каждого выполнения правила обновляется статус кампании.

5. Пример запроса на отключение РК:​

POST /api/v1/campaigns/disable
{
  "campaign_id": "12345",
  "platform": "Yandex",
  "action": "suspend"
}

Пример тела запроса:​

{
  "method": "suspend",
  "params": {
    "SelectionCriteria": {
      "Ids": ["12345"]
    }
  }
}

Таким образом, система использует таймеры через EventBridge и выполняет действия через Lambda, которые отправляют запросы на отключение/включение кампаний в API рекламных систем.

K50​

Detailed System Design for K50-Like Advertising Automation Platform​

Here is a breakdown of the architecture with specific details for scalability, functionality, and performance:

1. High-Level Architecture Components​

  1. User Interface Layer:

    • Dashboard: Centralized UI for campaign management, rule configuration, and analytics.
    • Admin Interface: For user account management and platform settings.
    • User Authentication & Permissions: Implemented using OAuth (for single sign-on) and AWS Cognito to manage access to user-specific campaigns.

  2. API Integration Layer:

    • Google Ads, Yandex Direct, Meta, VK, MyTarget APIs: Integration for pulling and pushing data. Each platform integration is handled by separate microservices.
    • Rate Limiting & Error Handling: Each API service includes logic for retrying failed requests and respecting the API rate limits of each platform.
    • Batch Data Sync: Daily synchronization to pull campaign data, using AWS Step Functions and AWS Lambda for scheduling and execution.

  3. Campaign Management:

    • Microservices-Based: Each service corresponds to a function (e.g., creating campaigns, updating budgets).
    • CRUD Operations: Implementing standard create, read, update, delete operations for campaigns using RESTful APIs.
    • Bulk Operations: For handling multiple campaigns (for instance, updating budgets for hundreds of campaigns at once), asynchronous jobs are used to prevent timeouts.

  4. Rule Engine:

    • Event-Driven: Rules (e.g., pause campaign when budget < X) are triggered by events (either on scheduled times or campaign changes).
    • Stored Conditions: In PostgreSQL or DynamoDB, rules are stored and checked daily.
    • AWS Lambda Functions: Executes the rules based on time or condition, e.g., check if a campaign needs to be paused, and triggers API calls to the ad platform.

  5. Data Storage:

    • PostgreSQL (Amazon RDS): Stores structured data related to campaigns, rules, users, and budgets. Optimized with indexing and partitioning for high-speed queries.
    • Amazon S3: Used for storing historical campaign data and large reports (e.g., up to 10 MB per report). Offers near-unlimited, cost-efficient storage for large files.
    • Amazon DynamoDB: Optionally used for storing rule executions, high-throughput write operations for rule triggering (due to its low-latency writes and scalability).
    • Redis: Used as a caching layer for active campaign data, reducing database load for frequently accessed data.

  6. Analytics & Reporting:

    • ETL (Extract, Transform, Load): Data from campaign reports is extracted daily and loaded into Amazon Redshift for analytics.
    • Dashboard Metrics: Provides real-time insights on active campaigns, budgets, and optimizations.
    • AWS Athena: Used to query historical data directly from S3 without moving it to the database. Allows ad-hoc querying over large datasets.

  7. Automation Engine:

    • Job Scheduler: AWS EventBridge triggers the automation engine at predefined intervals to check for campaigns needing updates (e.g., morning campaign sync, rule evaluation).
    • Worker Queues: When campaigns need updates across multiple platforms, jobs are sent to a distributed task queue (e.g., SQS) and processed by background workers (EC2 instances or Lambda).

  8. Scaling and Load Balancing:

    • Microservices: Deployed on Amazon ECS (Elastic Container Service) with auto-scaling to handle dynamic workloads.
    • Load Balancer: AWS Elastic Load Balancing (ELB) distributes traffic between the microservices. Helps manage peaks in API calls during campaign updates.
    • Horizontal Scaling: Services like API fetch, rule execution, and data retrieval are horizontally scalable via AWS services.

  9. Security:

    • IAM (Identity & Access Management): Configures permissions for accessing API keys, services, and ad platform credentials securely.
    • VPC (Virtual Private Cloud): The microservices and data layers operate inside a VPC to ensure isolation and security.
    • Encryption: Data at rest in PostgreSQL and S3 is encrypted using AWS KMS (Key Management Service), and communications between services are secured via HTTPS.

2. Data Flow​

  1. Client Logs into Dashboard:

    • User authenticates via OAuth or SSO using AWS Cognito.
    • The dashboard is loaded with data (e.g., active campaigns, rules) retrieved from RDS (PostgreSQL) or Redis (cache).

  2. Daily Campaign Sync:

    • AWS EventBridge triggers AWS Lambda at 6:00 AM daily.
    • Lambda function invokes API calls (Google Ads, Yandex Direct, etc.) through dedicated microservices to sync campaign data for all clients.
    • Data fetched is processed and updated in PostgreSQL. Historical snapshots of the data are archived in Amazon S3.

  3. Rule Execution:

    • Rules are evaluated after campaign sync, and Lambda checks the conditions stored in RDS/DynamoDB.
    • If a rule condition is met (e.g., budget exceeded), a call is made to the appropriate advertising platform’s API to pause or update the campaign.

  4. Analytics & Reporting:

    • Data is processed by an ETL job, which loads relevant data into Amazon Redshift.
    • Users query and generate reports from historical campaign data using Amazon Athena, directly accessing data from S3.

3. API Endpoints​

EndpointMethodDescription
/api/v1/campaignsGETFetch all campaigns for a client
/api/v1/campaigns/{id}POSTCreate or update a campaign
/api/v1/rulesPOSTCreate a rule for campaign management
/api/v1/rules/executePOSTTrigger execution of rules manually
/api/v1/analytics/reportsGETGenerate a report for campaign performance
/api/v1/integrations/syncPOSTTrigger manual synchronization with ad platforms

4. Non-Functional Requirements​

  • Scalability: Horizontal scalability with ECS and Lambda for scaling microservices automatically during high load (e.g., campaign syncing).
  • Fault Tolerance: Automatic retries on failed API calls, failover handling using AWS SQS for queuing API requests that encounter issues.
  • Data Consistency: Eventual consistency is acceptable for most operations, particularly between historical data and real-time campaign status.
  • Availability: Target 99.9% uptime with failover mechanisms and redundancy across multiple AWS Availability Zones.

5. Cost Optimization​

  • Lambda & Step Functions for event-driven processes reduce the need for always-on infrastructure.
  • S3 for cost-effective long-term data storage of large reports and historical data.

6. Technology Stack​

ComponentTechnology
API IntegrationPython/Node.js, AWS Lambda, API Gateway
Data StorageAmazon RDS (PostgreSQL), Amazon S3, Amazon DynamoDB
Queue & Async ProcessingAWS SQS, AWS Step Functions
AnalyticsAmazon Athena, Amazon Redshift, AWS Glue (ETL)
UI and FrontendReact.js/Angular, served via AWS Amplify or S3 + CloudFront
SecurityAWS Cognito, AWS IAM, KMS, VPC
DeploymentAWS ECS (Dockerized services), CloudFormation, CodePipeline

7. Scaling Strategy​

  • Auto-scaling groups: For dynamically adjusting the number of instances in the ECS cluster as campaign data fetch loads increase.
  • Global Data Storage: Use AWS Global Accelerator to route traffic efficiently from different regions.

This design ensures high scalability, security, and robust performance for a K50-like advertising automation platform.