Announcement

Collapse
No announcement yet.

Cloud logging and analysis

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Cloud logging and analysis

    What are some good reference materials around logging in cloud environments? Specifically getting Azure//GCP//AWS logs into a SIEM or collector, and building analysis queries around those logs?

    Splunk has builtin setup for pulling in AWS logs from various AWS applications like cloudwatch, but not the raw Cloudtrail logs, and I can't find any good documentation on an easy way to ingest those.

  • #2
    I haven't used Splunk in a long time, but for SumoLogic, we put the cloudtrail logs into S3 and they are shipped over to Sumo from there. One of the people on my team did the actual setup. I think there might have been a lambda job involved to help make the log shipping happen, but I'm not certain. I'll see if I can find out. Maybe something similar could work for Splunk.
    -AC

    ----
    Twitter: @AccidentalCISO
    Blog: https://www.accidentalciso.net/

    Comment


    • #3
      Does https://docs.splunk.com/Documentatio...AWS/CloudTrail cover what you're trying to do? Or is it missing something?
      I'm big on Splunk, but don't use AWS.

      Comment


      • #4
        accidentalciso - I'd love to hear what they have to say. We haven't settled on which SIEM we would use.
        Dzag - that helps. What I found when I first looked at it was a different set of recommendations. Mucho Gracias.

        Comment


        • #5
          I wouldn't call SumoLogic a SIEM, since it's missing the ticket tracking component to document the incidents. It works well enough to fire off log based alerts to a ticket system though, and you can build some pretty cool dashboards with it. Just don't fall for their "we're a SIEM replacement" spiel.

          I forgot to check on how the logs are shipped from S3 to Sumo. I'll make a note to myself to confirm on Monday.
          -AC

          ----
          Twitter: @AccidentalCISO
          Blog: https://www.accidentalciso.net/

          Comment


          • #6
            I've got a lot of customers who have worked on their SIEM integrations with a number of cloud services with some being extremely expensive to implement and maintain. Before going too far down that route, I would suggest you start defining very clearly
            • What data are you pulling and for what purpose?
              Most of the time, this is for some forensic analysis in a SIEM, but it may be part of a larger data lake for mining.
            • How much data are you pulling?
              Volume will matter.
            • Who will perform the analyses of the data?
              Writing and maintaining queries for cloud service data can get complicated and do not usually transfer easily across cloud services.
            • Who will consume the data or the analysis of that data? How often?
              Anyone who claims to review every report every day is likely being untruthful with themselves or others.
            • Is the data subject to statutory and regulatory controls (i.e., GDPR, CCPA)?
              These apply more than you may realize to audit data. Email addresses (e.g.) are transparent PII data.
            • For how long will you maintain the data? Is there a regulatory/statutory/policy requirement?
              Volume * Duration * Storage Cost + Compliance costs
            • After analyzing data, can I store it outside my SIEM less expensively, if I need to keep it?
              If you (e.g.) generate 10TB of audit data per day, you likely won't want to keep this in expensive storage for analysis.
            • How much will warm and cold storage of the data cost?
              Even cold storage cloud solutions get to be pricey at scale. TBs of data per week * weeks to store * Retrieval events = ?
            • How much will data being actively analyzed cost to process?
              Large ELS systems start to add up. Trust me on this one.

            Comment


            • #7
              Rapid7 IDR has some pretty easy to use integrations for AWS. Off the top of my head I think Cloudtrail and GuardDuty both forward to that. Plus Rapid7 is amazing with the Insight platform for small teams.

              Comment


              • #8
                Originally posted by accidentalciso View Post
                I haven't used Splunk in a long time, but for SumoLogic, we put the cloudtrail logs into S3 and they are shipped over to Sumo from there. One of the people on my team did the actual setup. I think there might have been a lambda job involved to help make the log shipping happen, but I'm not certain. I'll see if I can find out. Maybe something similar could work for Splunk.
                OK, so I checked with the team... I was close but not quite. Cloudtrail and other AWS logs do get placed in an S3 bucket, but there is no lambda job to ship off to sumoloic. Sumo is granted access to the bucket and consumes them from there. There are a number of "apps" in sumoloic that you can set up in your account to parse the logs and extract fields, etc... to make them more usable.
                -AC

                ----
                Twitter: @AccidentalCISO
                Blog: https://www.accidentalciso.net/

                Comment


                • #9
                  Karl Miller This is a good list. Thankfully I have already done this at $dayjob, and am aware of the cost impact.
                  accidentalciso In the short term the company is small enough not to need ticket tracking. If it gets to that point, a MSSP will be involved for that 24x7x365 eyes on glass. That leaves my options open.
                  indigocarmen2 I hadn't considered R7 IDR. I would be interested in seeing what they have to say, assuming I can set things up myself and minimize contact with sales staff. As the lady said, "Ain't no one got time fo' that."

                  Comment


                  • #10
                    Originally posted by 000Zero000 View Post
                    accidentalciso In the short term the company is small enough not to need ticket tracking.
                    Tickets serve as documentation, the need for which is not determined by org size.

                    If there isn't a ticket, the work never happened. It is invisible, anecdotal. You have no way to audit it, report on it, look back at information/solutions that resolved the issue, determine team performance, etc...

                    As the org grows, you are going to want that ticket system data to justify headcount.

                    Small orgs need tickets just as much as big orgs.
                    -AC

                    ----
                    Twitter: @AccidentalCISO
                    Blog: https://www.accidentalciso.net/

                    Comment

                    Working...
                    X