Types of SIEM Logs & Formats Explained | Comprehensive Guide

Security Information and Event Management (SIEM) systems are critical for monitoring and analyzing security events across an IT infrastructure. A SIEM aggregates, normalizes, and analyzes logs from various sources, helping organizations detect and respond to potential threats. Understanding the different types of logs and their formats is key to effectively leveraging a SIEM. This article delves into the common log types and their formats used in SIEM systems.

What Are Logs in SIEM?

Logs are records of events that occur within an IT environment. They are generated by devices, applications, servers, and networks, providing valuable insights into the system's health, activities, and potential security incidents. SIEM systems collect these logs and correlate them to identify patterns or anomalies that indicate security threats.

Different Types of Logs in SIEM

1.System Logs (Syslogs)

• Source: Operating systems (e.g., Windows, Linux, Unix).

• Purpose: Syslogs track system-level events such as user logins, logouts, file changes, and errors. They provide a foundational view of what’s happening on a system, helping to detect failures, errors, or unexpected behavior that could indicate security incidents.

• Log Content: Includes timestamps, severity levels, event descriptions, and often process or service IDs.

• Format Example:This follows the Syslog format (RFC 5424) and contains information like the priority value (<34>), timestamp, hostname, appname, event source, and detailed message.

  
  <34>1 2025-01-22T10:30:00Z hostname appname 1234 ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entr
  

2. Security Logs

• Source: Firewalls, intrusion detection/prevention systems (IDS/IPS), antivirus solutions, and access control systems.

• Purpose: Security logs record events related to security threats such as blocked connections, intrusion attempts, malware detections, and unauthorized access. These logs are crucial for detecting and responding to potential cyberattacks or policy violations.

• Log Content: Includes source and destination IPs, action taken (e.g., blocked, allowed), protocols, and rule identifiers.

• Format Example:This JSON example includes all the relevant details about the event, including the action taken and the involved IP addresses.

  {
    "timestamp": "2025-01-22T10:45:00Z",
    "source_ip": "192.168.1.10",
    "destination_ip": "10.0.0.5",
    "action": "blocked",
    "protocol": "TCP",
    "rule": "Block_Malicious_IP"
  }
  

3. Application Logs

• Source: Web servers (e.g., Apache, Nginx), databases, business applications, and cloud services.

• Purpose: Application logs capture events specific to an application’s behavior, such as errors, user actions, and performance metrics. They help developers and system administrators monitor application health and identify faults.

• Log Content: These logs can contain timestamps, log levels (e.g., ERROR, INFO), and messages detailing specific actions or issues within the application.

• Format Example:This entry records an error in a specific application module, providing a timestamp, severity level, and error message for quick troubleshooting.

  2025-01-22 10:50:00,123 [ERROR] app.module - Error occurred in module X: NullPointerException
  

4. Network Logs

• Source: Routers, switches, firewalls, and other network devices.

• Purpose: Network logs capture information about network traffic, including routing updates, connection attempts, and successful or failed communications. These logs are essential for identifying network anomalies, bottlenecks, and security incidents like DDoS attacks or unauthorized access.

• Log Content: They include details such as IP addresses, ports, protocols, and actions taken (e.g., allowed or denied).

• Format Example:This log records an access control list (ACL) rule being applied to allow traffic from one IP to another, including the protocol and port.

  Jan 22 10:55:00 192.168.1.1 %SEC-6-IPACCESSLOGP: list ACL-10 permitted tcp 192.168.2.10(12345) -> 10.0.0.5(80), 1 packet
  

5. Audit Logs

• Source: Operating systems, applications, databases.

• Purpose: Audit logs track changes to system configurations, permissions, and data access. They are key for compliance audits and for tracking user actions on critical resources (e.g., who changed a password or accessed sensitive data).

• Log Content: They often include the user performing the action, the action itself (e.g., creating, modifying, or deleting data), and the result (e.g., success or failure).

• Format Example:This JSON entry tracks a successful password change by an administrator on a user account.

  {
    "timestamp": "2025-01-22T11:00:00Z",
    "user": "admin",
    "action": "changed_password",
    "target": "user_account",
    "result": "success"
  }
  

6. Endpoint Logs

• Source: Laptops, desktops, mobile devices, and endpoint protection software.

• Purpose: Endpoint logs capture activity on individual devices. They help monitor endpoint security, track file accesses, software installations, and other user activities. This is crucial for detecting malware, unauthorized changes, or unusual behavior on endpoints.

• Log Content: Includes timestamps, device IDs, the type of event (e.g., file access), and details of the action.

• Format Example:This log tracks a user reading a sensitive document on a device, which could be monitored for security reasons.

  {
    "timestamp": "2025-01-22T11:15:00Z",
    "device_id": "XYZ12345",
    "event": "file_access",
    "file_path": "C:\\\\\\\\Sensitive\\\\\\\\Document.pdf",
    "action": "read"
  }
  

7. Cloud Service Logs

• Source: Cloud platforms like AWS, Azure, and Google Cloud.

• Purpose: Cloud service logs provide insights into activities and events within cloud environments, such as resource provisioning, API calls, and user access. They help administrators ensure that cloud resources are being used according to policy and can also highlight potential security risks.

• Log Content: Includes resource names, event types (e.g., StartInstances), user identities, and geographic regions.

• Format Example:This log entry captures an EC2 instance being started in AWS by a specific user.

  {
    "timestamp": "2025-01-22T11:30:00Z",
    "service": "EC2",
    "event_name": "StartInstances",
    "user": "jdoe@example.com",
    "region": "us-east-1",
    "instance_id": "i-0abcd1234efgh5678"
  }
  

8. Database Logs

• Source: SQL Server, Oracle, MySQL, and NoSQL databases.
• Purpose: Database logs record activities such as queries, user actions, and performance metrics (e.g., query execution time). These logs help in performance tuning, auditing database access, and tracking potential misuse or abuse.
• Log Content: Includes details such as query information, user identity, and performance-related metrics.
• Format Example:This log shows an admin user querying the database for specific customer information and the duration of the query execution.

Why Log Formats Matter in SIEM

Standardized log formats ensure that SIEM systems can efficiently parse and analyze data from diverse sources. Common log standards include:

• Syslog (RFC 5424): Widely used for system and network logs.
• JSON: Common for structured logs from modern applications and APIs.
• CEF (Common Event Format): Vendor-neutral log format for security events.
• LEEF (Log Event Extended Format): Lightweight format designed for security logs.
• Custom Formats: Proprietary formats used by specific vendors or applications.

Best Practices for Managing Logs in SIEM

1. Centralize Log Collection: Use log collectors or forwarders to aggregate logs from all sources.
2. Normalize Log Data: Ensure logs are converted into a consistent format for easy analysis.
3. Enable Detailed Logging: Configure systems and applications to generate verbose logs for better visibility.
4. Monitor Log Storage: Regularly manage log retention policies to balance storage costs and compliance needs.
5. Leverage Correlation Rules: Use SIEM correlation rules to detect patterns and anomalies across different log types.

Conclusion

Understanding the different types of logs and their formats is essential for effective SIEM operation. By aggregating and analyzing logs from various sources such as systems, applications, networks, and endpoints, SIEM systems provide invaluable insights for detecting and responding to security threats. Adhering to log management best practices ensures a robust and efficient security monitoring process.