Cybercrime is big business, and the implications for safety and security encompass individuals, governments, educational institutions, and companies of all sizes. Additionally, the economic impact of global cybercrime is enormous. According to Cybersecurity Ventures and Cybercrime Magazine, global cybercrime costs are expected to grow by 15% per year over the next five years, reaching $10.5 trillion USD annually by 2025, up from $3 trillion back in 2015. This represents (according to Cybersecurity Ventures) “the greatest transfer of economic wealth in history.”
Many tools and techniques are available to help detect and deter criminals and hackers from gaining access to your IT environment. One important tool is the "honeypot," a virtual trap used to lure attackers. A honeypot is a system, device, or software that is intentionally compromised to expose opportunities for attackers so they can then be studied to improve security policies.
Researching Kubernetes Deployments with Helix Honeypot
Software architecture has changed dramatically over the past few years. We once relied upon dedicated application servers in data centers. Then, virtualization technology became a huge enhancement that reduced the number of physical machines and created significant cost savings. And now, containerization is the next evolution of virtualization, spawning open source projects that aid in the operation of containers and containerized environments – the most popular being Kubernetes.
Despite the popularity, research on public-facing Kubernetes deployments is scarce. Currently, deploying a honeypot for Kubernetes requires a full implementation which leads to additional compute and management overhead. Helix Honeypot was created to help solve some of the pain points when doing threat research around public-facing Kubernetes deployments.
Kubernetes Overview
Kubernetes typically ships as a cluster of components. These components live on worker machines called nodes. Nodes will host all of the production workload depending on where Kubernetes is deployed. A part of the workload that ships with Kubernetes is called the control plane.
The control plane is made up of multiple services that control and manage the lifecycle of containers. The API server is the frontend of the control plane and is responsible for exposing the Kubernetes API. Helix Honeypot was created to expose and mock this service on the public internet.
Helix Honeypot 30-Day Deployment
A few members of the Automox security team decided to run a 30-day test of Helix Honeypot. We wanted to deploy Helix Honeypot as easily and as inexpensively as possible so we opted for free-tier AWS instances and shipped logs to a centralized solution for storage and analysis.
Additionally, using Docker allowed us to work quickly and in a relatively easy manner. Docker is an open platform for developing, shipping, and running applications. It enables you to easily separate your applications from your infrastructure so you can deliver software faster.
Helix was built using golang and the echo framework. This allows for easy dockerization and deployment of the API without the typical K8s control plane components.
We built and shipped the container to Dockerhub “helixhoneypot/helixhoneypot:latest” and then hosted it on cloud instances using docker-compose to manage the services.
Logs were aggregated into a Graylog instance using the GELF docker driver, as the honeypot only supports writing to standard output as of now.
After the deployment, the service listened and responded like a typical K8s API.
Findings from our Honeypot Experiment
Our expectation was that within 30 days a malicious payload would be deployed containing some form of cryptomining, like XMRig.
After 30 days, however, only basic GET requests were made to a few API endpoints. Two of the endpoints typically store key value data that may contain sensitive information about the workloads deployed. The last is just a basic information-gathering endpoint about the pods deployed.
The requests came in a daily cadence from multiple IPs and User Agents and are no different than the vulnerability scanners you typically see scanning the internet. The data shows two distinct scanners:
- One scanning for “pods and secrets,” and
- One scanning for “pods, secrets, and configmaps.”
No POST requests were observed on the honeypot during the 30-day window.
Outlined below are snapshots of the data pertaining to each endpoint:
/api/v1/configmaps
/api/v1/pods
/api/v1/secrets
Our Takeaways: Next Steps in the Honeypot Evolution
Expanding the honeypot to expose a kubelet-like service would be a great next step in the honeypot’s evolution. Additionally, some endpoints didn’t respond correctly when doing certain operations, such as creating a single pod. A typical response would be a json payload containing the status of the pod created. Our honeypot seemed to return a value not supported by kubectl.
It would also be valuable if honey tokens – credentials or secrets that are intentionally leaked to see who and what uses them – could be specified and returned from the secret’s endpoint. This information could be used to further harden your company's security posture and alert you if you are being actively probed.
The full source code for Helix Honeypot is available here.
Staying Vigilant...and Proactive with Vulnerability Management
While a honeypot may be an effective tool to lure hackers in so their exploitation attempts can be studied, the most effective security policy on production environments is to prevent cybercriminals from gaining access in the first place.
Currently, the average time to fix high-severity vulnerabilities can be as high as 246 days for enterprises. Comparatively, criminals and bad actors weaponize critical vulnerabilities in an average of seven days, which is why Automox recommends following the “24/72 Threshold for Endpoint Hardening.” This simple benchmark states that critical vulnerabilities must be patched within 72 hours and zero-day vulnerabilities within a 24-hour window.
Learn More
- How can I automate my cybersecurity compliance?
- How do I get started with Kubernetes?
- Can an Automox Worklet update Docker Containers on Linux?
About Automox IT Operations
Today’s IT leaders deserve better than tedious legacy tools to manage their infrastructure. From our single cloud-native platform, automate and scale your IT operations to meet the growing business demands of the modern workforce. With complete visibility of your entire environment, you can easily monitor, identify, and respond to issues in real-time across any endpoint, regardless of OS or location.
Demo Automox to see how you can immediately gain effortless command of your endpoints.
