In a London apartment, Shay Banon was looking for a job while his wife attended cooking school at Le Cordon Bleu. In his spare time, he started building a search engine for her growing list of recipes.
His first iteration was called Compass. The second was Elasticsearch (with Apache Lucene under the hood). He open sourced Elasticsearch, created the #elasticsearch IRC channel, and waited for users to appear.
The response was impressive. Users took to it naturally and easily. Adoption went through the roof, a community started to form, and people noticed — namely Steven Schuurman, Uri Boness, and Simon Willnauer. Together, they founded a search company.
Around the time Elasticsearch Inc. was founded two other open source projects were taking flight.
Jordan Sissel was working on Logstash, an open source, pluggable ingestion tool for sending logs to the "stash" of the user’s choosing, one of which was Elasticsearch. He was also developing a UI on top of it to visualize log data — and it was shaky at best.
Fortunately, someone else was tinkering with the visualization challenge. Enter Rashid Khan, who was working on an open source UI called Kibana.
Shay, Jordan, and Rashid had known each other and their projects for some time and decided to team up, resulting in the ELK Stack — Elasticsearch, Logstash, and Kibana Stack.
A while later, we released two commercial plugins: Marvel for monitoring and Shield for security.
At Elastic{ON} 2015 in San Francisco, we made two big announcements. The first: we rebranded the company to be called Elastic. The new name better represented our growing product ecosystem and suite of use cases. The second: we joined forces with Found, a company delivering hosted and managed Elasticsearch on AWS. By teaming up, we could provide the simplest, most complete offering on the market.
In the early days, building and releasing software at Elastic took an every-engineer-for-themselves approach: ship whatever version you want, whenever you want — just make it awesome. Kibana had betas, Logstash had milestones, Elasticsearch had numbers. Plugins happened as they pleased. It was chaotic, but it worked...until it didn't.
As users were doing more with the product, we needed to build a product that did more for the users. We added more capability, submitted more pull requests, built new plugins and extensions. The awesomeness increased, complexity emerged, and things got messy for our technology stack.
For instance, if you were running version 1.7 of Elasticsearch and version 2.3 of some plugin, there wasn’t an automatic way to know if they were compatible or if the plugin was silently failing. This was a bug.
We also started to hear ourselves say things like, “If you want to use Shield, you need Elasticsearch 1.4.2….unless you’re using Watcher. In that case, you’ll need Elasticsearch 1.5.2. And if you’re using Elasticsearch 1.5.2, that’s only compatible with Kibana 4.0.x, Logstash 1.4.x, Shield 1.2.x, and Watcher 1.0.x.”
We had arrived at a special sort of versioning hell — and the support matrix didn’t look any better. It was time for a change.
While the product teams wrestled with version numbers, another product story was unfolding. In 2015, we welcomed Packetbeat, a Berlin-based, husband-and-wife team engineering a lightweight way to send network data to Elasticsearch, to the Elastic family.
That got us thinking: what if we had a family of single-purpose, lightweight data shippers to send network data, logs, metrics, audit data, and more from edge machines to Logstash and Elasticsearch? And so Beats was born.
October 2015 marked a turning point for addressing our product versioning and compatibility complexities.
Dubbed the “release bonanza,” it was the first time all of our products — Elasticsearch 2.0, Logstash 2.0, Watcher 2.0, Shield 2.0, and Kibana 4.2 — shipped together on the same day. (Beats 1.0 had another month to bake.)
Coordinating this effort wasn’t easy. The engineering teams had to change the way they worked together to build and test the products. It was worthwhile, though. This shift made it easier for users to get started with our products and made our products more reliable to do amazing things with.
Aligning release cadence with Elasticsearch 2.0 was the first step toward a more mature product offering. The 5.0 launch was the second step. It introduced a more integrated, better tested, and easier getting started experience than ever before.
The 5.0 release also bundled all of our commercial plugins (which we called Shield, Marvel, and Watcher at the time) into a single extension called X-Pack. It consisted of features like security, monitoring, and alerting for our core products, and grew to include machine learning when we brought a London-based company called Prelert into the Elastic family.
In version 5.3 (released in March 2017), Filebeat formally introduced the concept of "modules," or a set of safe configurations to ship, parse, store, analyze, and visualize common log formats (e.g., Apache, Nginx, MySQL, etc.) in the Elastic Stack. Modules simplified the getting started experience of going from dataset to dashboard.
Metricbeat and Packetbeat had their own flavors of modules, and months later, Logstash would introduce modules of its own for ArcSight and NetFlow data.
From the beginning, we had a vision for simplifying how users deploy Elastic in their organizations. We took the technology we use to manage our own Elastic Cloud service and released Elastic Cloud Enterprise (or ECE) to allow businesses big and small to download all the goodness of our hosted offering and run it themselves. ECE made managing one cluster or thousands seamless, streamlining the management and orchestration of Elastic products and solutions in any environment.
As modules started to multiply, getting started with the Elastic Stack to address a particular use case like logging or metrics got easier and easier. And momentum continued to build when we joined forces with Opbeat, a Copenhagen-based application performance monitoring (APM) company, and Swiftype, a San Francisco-based site and enterprise search company, a few months later. Both companies became part of Elastic.
By this time, our company had matured to a place where we could offer streamlined ways for solving common problems leading us to formally introduce our solutions. While our solutions range from a DIY experience to something more turnkey, each has real product behind it and can be deployed in just a few minutes.
From open source to open communication, being open is at the heart of all we do. This is why we made the decision to open the code to our commercial X-Pack features in order to speed up development time and increase community engagement, allowing everyone to contribute to, comment on, and inspect our code.
As a result, getting started with the Elastic Stack became even easier, with all of the X-Pack features now shipping with the default distributions of Elasticsearch, Kibana, Beats, and Logstash. This change didn’t take away any Apache 2.0 code. Instead, we doubled down on being open.
In July, we acquired Insight.io, a Palo Alto-based startup that developed search tools that provide a semantic understanding of software source code. Built on the Elastic Stack, Insight.io gives developers the ability to search for code on specific application functionality and dependencies, providing IDE-like code intelligence features such as cross-reference, class hierarchy, and semantic understanding. Basically, it expanded our search capabilities, making life easier for developers. (And for organization leaders!)
At exactly 9:30 a.m. Eastern on October 5, the bell at the New York Stock Exchange rang out, officially making Elastic a public company. With a record-setting 230 Elasticians on the trading floor and hundreds more around the world, our distributed company celebrated reaching this remarkable milestone together. While it is just one day in our long journey, it was a pretty grand one.
In June 2019, we launched Elastic SIEM to give security teams a powerful new tool for collecting, investigating, and detecting security information and events. We also acquired Endgame, a pioneer and industry-recognized leader in endpoint protection, detection, and response. Endgame's endpoint protection along with Elastic SIEM further simplified security for everyone — giving users a single, complete solution to combat threats and attacks.
With teams around the world working and collaborating remotely, we announced the general availability of Elastic Workplace Search. Easily implemented with out-of-the-box connectivity to SaaS and cloud-based data sources, indexing content from a modern toolchain, it offers a relevant, personalized, and modern unified search experience. Basically, it's a centralized way to search the knowledge scattered across all the applications your team uses every day. And it's awesome.
We held our first annual Elastic Community Conference (ElasticCC) on February 26 and 27. The free, global technical conference offered success stories, tips, lessons, and more from the community, for the community. We had fun and nerdy discussions in several languages, including English, Chinese, French, Korean, and Portuguese!
On August 3, we launched the industry’s first free and open Limitless Extended Detection and Response (XDR). Part of Elastic Security, Elastic Limitless XDR modernizes security operations by unifying the capabilities of security information and event management (SIEM), security analytics, and endpoint security on one platform. All that, and you can centrally manage it in Kibana.
Protecting the world’s data and systems from attack is a big job, but we made it easier by joining forces with build.security. Acquiring the Tel Aviv-based company on August 23 helped us move toward cloud security enforcement on hosts, virtual machines, and containers orchestrated by Kubernetes.
After acquiring build.security, we joined forces with infrastructure detection and response leader Cmd, giving our customers cloud security protections from build-time to deployment-time to runtime. By adding the capabilities of Cmd's expertise and product into Elastic Security, customers can detect, prevent, and respond to attacks on their cloud workloads.
In November, we accelerated our vision for unified, actionable observability and enhanced the ability to detect and find root causes faster in complex distributed environments. That's quite a mouthful to say that we acquired Optimyze. This "always-on" continuous profiling platform offers a simpler way to get insights into entire IT ecosystems and eliminate blind spots.
Throughout 2021 we were recognized in several analyst reports. We were named a Challenger in the 2021 Gartner Magic Quadrant for Insight Engines, a Visionary in the 2021 Gartner Magic Quadrant for Application Performance Monitoring, and a Leader in The Forrester Wave™: Cognitive Search (Q3, 2021). We were also recognized in the 2021 Gartner Magic Quadrant for SIEM and in the Forrester New Wave™: Extended Detection and Response (XDR) Providers, Q4 2021.
In 2021, we celebrated our 10th birthday. Before heading into our tween years, we noted that our community had grown to 1.5 million members, 263 global groups, and 150,000 who host our regular community meetups. We're honored to be a part of the global community and to continue to help everyone find anything and everything they need faster!
There's more to our story. Stay tuned for updates as our adventure continues to unfold.